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ABSTRACT 


This research reviews the two most widely used software 
capability evaluations in the Department of Defense: the Software 
Capability Evaluation (SCE) created by the Software Engineering 
Institute (SEI), and the Software Development Capability Evaluation 
(SDCE) created by the US Air Force’s Aeronautic Systems 
Command (ASC). Their use as part of the source selection 
evaluation of a contractor developing a software intensive weapon 
system is examined. The objective of this thesis is to describe each 
evaluation method, then highlight their respective strengths and 
weaknesses. The result is a guide that will assist Program Managers 
in deciding which software capability evaluation is more suitable for 
use in their program. 
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I. INTRODUCTION 


A. BACKGROUND 

In today’s environment of acquiring weapon systems within the Department of 
Defense (DOD), effective software management is vital to the success of a program. 
Modern weapon systems rely heavily on their computer hardware and software to operate 
in a manner in which they were designed [Ref. 1]. 

Several factors have led to the tremendous growth in the use of computer hardware 
and software in today’s weapon systems. The increasing capabilities of threat weapons, 
combined with a finite defense budget, led to DOD’s policy of emphasizing qualitative 
rather than quantitative weapons superiority. Advances in microprocessor and integrated 
circuit technology led to tremendous increases in the capabilities of computer hardware 
with corresponding decreases in size, power requirements, and costs. As performance 
requirements for new weapon systems increase, so does the number of on-board 
computers and the number of source lines of code (SLOC). [Ref. l:p. 2-2] 

The growth in SLOC found in weapons is illustrated by US Air Force (USAF) 
fighter aircraft. The Vietnam era F-4 Phantom fighter had virtually no software. Today’s 
F-16D Falcon fighter has approximately 236,000 SLOC. The next generation fighter, the 
Advanced Tactical Fighter (ATF), is projected to require 5,000,000 to 7,000,000 SLOC. 
[Ref. 2] Computers and software have improved weapon system performance and 
have expanded the capabilities of the human operator. 
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While mission-critical software has grown dramatically in complexity and 
magnitude, the systems and software engineering management discipline necessary to 
successfully develop this software has not kept pace. Program Managers (PMs) and 
defense contractors must meet the difficult task of developing large and complex software 
systems that are critical to system performance often without well-defined and 
consistently applied systems engineering, software engineering, and management 
discipline. [Ref. 3] Such shortfalls have adversely impacted many weapon system 
programs. Software development problems are well-documented in numerous General 
Accounting Office (GAO) reports [Ref. 4]. 

The most common software problems are development problems, which lead to 
program schedule slippages, cost overruns, and delivered software that does not meet the 
stated requirements [Ref. l:p. 1-1]. In addition, software production remains labor 
intensive. Throughout the late 1980s and early 1990, there was increasing concern within 
DOD and the defense industry that there would not be enough labor available to produce 
software to meet the increasing demands of both the US Government and the civilian 
sector. Because of these problems, software has been referred to by some as the "achilles 
heel" of DOD. [Ref. 2:p. 28] 

An important theme in Total Quality Management (TQM) is that the quality of a 
product is directly related to the process used to produce it. Through continuous process 
improvement, variance in manufacturing time and cost is reduced. The frequency of 
meeting customer requirements increases. [Ref. 5] Process improvement normally 
leads to an increase in productivity [Ref. 6]. For these reasons, attention has 
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focused on the process used to develop software as a key factor in addressing the 
problems facing software development today. 

The Software Engineering Institute (SEI) defines process capability as the inherent 
ability of a process to produce planned results [Ref. 7]. Planned results means 
meeting all requirements within time and cost restraints. In the past, contractors were 
selected through the source selection process based on factors such as past performance, 
technical approach, cost, and schedule. An important factor that was ignored during this 
source selection process was whether prospective vendors had the process capability (in 
terms of software engineering, systems engineering, and management controls) to execute 
their respective proposed programs as planned. The challenge facing today’s PMs is 
selecting the best contractor that not only has the capacity but also the capability to 
produce mission-critical software. [Ref. 3:p. 1] 

Capability evaluations are independent evaluations of a vendor’s software 
development process that identify its strengths and weaknesses as they relate to a 
particular acquisition [Ref. l:pp. 8-12 - 8-15]. Evaluations can be conducted to determine 
that a reasonable software process is practiced, documented, enforced, staffed by well- 
trained personnel, and measured. Capability evaluations have been used in DOD source 
selection since the late 1980’s [Ref. 15:p. 4]. The decision to use evaluations on a project 
was mainly left to the discretion of the PM. 

With the release of USAF Acquisition Policy Memorandum 93M-003 on June 4, 
1993, the USAF became the first Service to require the use of capability evaluations 
during source selection of software intensive projects which meet certain criteria. Two 
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capability evaluations are authorized for use by the USAF. They are the Software 
Capability Evaluation (SCE) developed by the SEI, and the Software Development 
Capability/Capacity Review (SDCCR) created by the Department of the Air Force 
Aeronautical Systems Center, Air Force Materiel Command (ASC/AFMC). 

Since June 4, 1993, both capability evaluations have been revised. In July 1993, 
SEI released SCE version 1.5. In November 1993, ASC/AFMC published the Software 
Development Capability Evaluation (SDCE), which supersedes the SDCCR. 

B. OBJECTIVES 

The objective of this thesis is to provide assistance to PMs in deciding which of the 
two software capability evaluations, SEI’s Software Capability Evaluation version 1.5 or 
ASC/AFMC’s Software Development Capability Evaluation, is better suited for use on 
their program during source selection for evaluating a contractor’s capability to develop 
a software intensive weapon system. To meet this objective, an overview of each 
capability evaluation method, together with their corresponding models, will be presented. 
The strengths and weaknesses of each evaluation will be analyzed. The result will be a 
guide that a PM can use in selecting between the two software capability evaluation 
methods. 

C. THE RESEARCH QUESTION 

The primary research question for this thesis is "What are the strengths and 
weaknesses of the SEI’s Software Capability Evaluation and ASC/AFMC’s Software 
Development Capability Evaluation, that PMs should consider when deciding which 
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method to use on their program during source selection to evaluate a contractor’s software 
development capability ?" In support of the primary research question, this thesis will 
address the following subsidiary questions: 

1. What is the Capability Maturity Model? 

2. What is a Software Capability Evaluation and how is it used during source 
selection? 

3. What is the Software Development Capability Evaluation Model? 

4. What is a Software Development Capability Evaluation and how is it used 
during source selection? 

D. SCOPE, LIMITATIONS, AND ASSUMPTIONS 

Acquisition Policy Memorandum 93M-003 applies only to the USAF. The USAF 
is the only Service to use the SDCE and its predecessor, the SDCCR. This thesis, 
therefore, only examines the USAF’s use of the SCE and SDCE during source selection. 

The use of capability evaluations is not limited to source selection only. Other uses 
include monitoring contractor performance after the contract has been awarded, selecting 
subcontractors by the prime contractor, and as input to selecting the appropriate metrics 
required to effectively measure the status of a program’s software development. This 
thesis, however, will only address the use of capability evaluations during source selection 
evaluations of a contractor. 

This thesis assumes that the reader is familiar with Government source selection 
procedures. Discussions concerning source selection organizations, responsibilities, 
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regulations, and procedures found within this thesis are limited to those areas affected by 
the capability evaluation process. 

E. LITERATURE REVIEW AND METHODOLOGY 

The research for this thesis was conducted in two steps. The first steps involved 
an intensive literary search of published material concerning both capability evaluations. 
The main effort focused on the written work of each organization on the concept and 
implementation of their respective evaluation method. 

For the Capability Maturity Model (CMM) on which the SCE is based, the primary 
SEI works are CMUISEI-93-TR-24 Software Engineering Institute, Capability Maturity 
Model (CMM) Software, Version 1.1 and CMUISEI-93-TR-25 Software Engineering 
Institute, Key Practices of the Capability Maturity Model, Version 1.1. The majority of 
the information for the SCE itself came from CMU/SEI-TR-17 Software Engineering 
Institute, Software Capability Evaluation (SCE) Version 1.5 Method Description. 

Information concerning the SDCE came primarily from AFMC Pamphlet 800-61, 
Acquisition Management, Software Development Capability Evaluation dated 24 
November 1993, which is published in two volumes. Other sources of information 
include articles from professional journals as well as books on software quality. 

The second aspect of this research involved conducting interviews to obtain answers 
to specific questions not addressed in any literature. An interview was conducted with 
the originator of the USAF’s acquisition policy to clarify the contents of the 
memorandum. Personnel from SEI and ASC/AFMC were interviewed about their 
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respective evaluation methods. Contractors were also interviewed to get an industry 
perspective on these evaluations. 

F. ORGANIZATION OF STUDY 

The remaining chapters of this thesis are organized into three major parts. The 
first part, consisting of Chapters II through V, will introduce the reader to the two 
capability evaluations being studied, and the models on which they are based. Chapter 
II describes the CMM. The next chapter outlines the procedures on how to conduct an 
SCR, The SDCE model is described in Chapter IV, while the actual SDCE activities are 
covered in Chapter V. Chapter VI presents the results of two studies. 

The second part involves a detailed comparison of both capability evaluations. 
Chapter VII compares each method in terms of key factors to highlight the strengths and 
weaknesses of the SCE and SDCE. 

Finally, Chapter VIII concludes with a summary of the strengths and weaknesses 
of each evaluation. This may serve as a guide for PMs in determining which evaluation 
to use during source selection evaluation for their program. 


7 


n. THE CAPABILITY MATURITY MODEL 


This chapter introduces the reader to the Capability Maturity Model (CMM) version 
1.1. The SCE measures a contractor’s strengths, weaknesses, and improvement activities 
against the principles of the CMM. It is therefore important for the reader to understand 
the CMM in order to understand the SCE. This chapter provides an overview of the 
CMM. A more detailed description of the model can be found in the SEI publication, 
CMU/SEI-93-TR-24 Software Engineering Institute, Capability Maturity Model (CMM) 
Software, Version 1.1. 

A. DEVELOPMENT 

The USAF established the SEI in December of 1984, an affiliate of the Camegie- 
Mellon University in Pittsburgh, Pennsylvania, under contract as a Federally Funded 
Research and Development Center (FFRDC). The SEI’s mission is to provide leadership 
in advancing the state of the practice of software engineering to improve the quality of 
systems that depend on software. [Ref. 8] It was tasked with researching the 
transition of new software technology, analyzing software development environments, and 
providing education in the software and system engineering process. [Ref. l:p. 4-7] SEI’s 
primary vision is to bring engineering and discipline to the development and maintenance 
of software [Ref. 8]. It is this vision that led to the creation of the CMM. 
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In November 1986, SEI, with the assistance of the MITRE Corporation, began 
developing a software process maturity model to create a conceptual structure for 
improving the management and development of software products in a disciplined and 
consistent way. SEI released a brief description of the model in September 1987. It was 
later expanded in Watts Humphrey’s book, Managing the Software Process. SEI 
developed a maturity questionnaire, a set of "yes/no" questions that address organization 
and management issues, based on this model. 

Over the next few years, SEI created two methods for using the questionnaire and 
the maturity model. The Software Process Assessment (SPA) provides the means for an 
organization to perform self-audits to identify its strengths, weaknesses, existing 
improvement activities, and areas for improvement. The second assessment method, the 
Software Capability Evaluation (SCE), is a tool used by an outside agency to evaluate an 
organization’s software process capability. The results are used during source selection. 
Feedback on the software process maturity model was incorporated in CMM version 1.0, 
released in August 1991. The current version of the CMM, version 1.1, was released in 
February 1993, and is the result of comments from an April 1992 workshop and ongoing 
feedback from the software community. [Ref. 7;pg. 18-19] 

The CMM does not guarantee that software products will be successfully built or 
that all problems in software engineering will be adequately resolved [Ref. 8:p. 135]. It 
gives developers a tool to gain control of their software development process and move 
towards continuous process improvement. It focuses on a set of key process areas that 
have been proven to enhance software development and maintenance capability. The 
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CMM does not address every issue concerning the software development process and 
quality improvement. Issues that are not directly addressed include specific tools, 
methods and technologies, concurrent engineering and teamwork, systems engineering, 
human resources, change management, and expertise in a particular domain 
[Ref. 9]. By focusing on a limited set of activities and working aggressively to 
achieve them, a developer can steadily improve its process capability [Ref. 8:p. 7]. 

B. STRUCTURE 

The CMM resembles a hierarchial structure. The highest level is the maturity level. 
SEI defines a maturity level as a well-defined evolutionary plateau on the path towards 
becoming a mature software organization. There are five maturity levels, with the lowest 
being Maturity Level 1. Each level serves as a foundation for the subsequent maturity 
level. [Ref. 7:p. 20] 

SEI claims that as an organization’s maturity increases, three types of improvements 
can be expected. First, the difference between planned results and actual results decreases 
across projects. Second, the variability of actual results around targeted results decreases. 
Finally, as an organization matures, cost and development time decrease while 
productivity and product quality increase. SEI admits that there are no definitive studies 
confirming these claims. These expectations are based on quantitative results of process 
improvements achieved by organizations using the SPA and the CMM. [Ref. 10:p. 3-54] 
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With the exception of Maturity Level 1, each maturity level is composed of several 
key process areas (KPA). KPAs are groups of related activities that, when performed 
collectively, achieve a set of goals considered important for enhancing process capability. 
They identify the issues that must be addressed to achieve a specific maturity level. [Ref. 
7:p. 26] 

Each KPA has a set of goals that specify the KPA’s scope, boundaries, and intent. 
They are used to determine whether an organization has effectively implemented a key 
process area. [Ref. 7:p. 26] 

KPAs are organized into five sections called common features. The common 
features consist of attributes that indicate whether the implementation and 
institutionalization of a key process area are effective, repeatable, and lasting. The five 
common features are: 

• Commitment to Perform - Describes actions the organization must take to ensure 
the process is established and will endure. This typically involves establishing 
organizational policies and obtaining senior management commitment. 

• Ability to Perform - Provides the preconditions that must exist within the 
organization to implement the software process competently in terms of resources, 
organizational structure, delegation of responsibility, and training. 

• Activities Performed - Conveys the roles and procedures required to implement 
a KPA. It involves establishing plans and procedures, performing the work, tracking the 
work, and taking corrective actions as required. 
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• Measurement and Analysis - Describes the need to measure the process and 
analyze the results. This typically involves obtaining sample measurements to determine 
the status and effectiveness of the activities being performed. 

• Verifying Implementation - Provides the steps necessary to ensure all activities 
are performed in compliance with the established process. These steps include reviews 
and audits conducted by senior management, project management, and software quality 
assurance personnel. [Ref. 7:p. 26] 

The common features contain key practices that describe the infrastructure and 
activities that contribute most to the effective implementation and institutionalization of 
a key process area. They consist of a single descriptive sentence, and often include 
subpractices that describe what should occur for the key process to be implemented 
satisfactorily. Supplementary information, which contains examples, elaborations, and 
references to other KPAs, is also provided. While key practices describe what should be 
done, they do not mandate how to do it. [Ref. 7:p. 26] 

C. MATURITY LEVELS 

As previously mentioned, maturity levels are a well-defined evolutionary plateau 
on the path towards becoming a mature software organization. They are used as a means 
of characterizing an organization’s process capability. 

1. Level 1: Initial 

SEI characterizes this initial level as immature. An immature organization normally 
does not provide a stable environment for developing and maintaining software. 
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Processes are ad hoc and often improvised. These organizations frequently have difficulty 
meeting commitments. This frequently results in a series of crises during which planned 
procedures, especially testing and reviews, are reduced in scope or abandoned. [Ref. 7:p. 
19] 

Immature organizations do deliver software that meets all requirements, but they are 
usually behind schedule and over budget. For Level 1 organizations, success is dependent 
on the talents and efforts of individuals. Continued success requires using the same 
development team on new projects or the constant recruitment of highly skilled and 
competent personnel. Occasionally, good software managers can instill a disciplined 
approach to software development. However, when they leave their stabilizing influence 
leaves with them. SEI characterizes Level 1 as unpredictable. The main focus is on the 
talents of individuals. [Ref. 7:pp. 21-22] 

2. Level 2: Repeatable 

At the repeatable level, the organization has established basic policies and 
implementation procedures for managing a software project. Realistic project planning 
and effective management controls are the result of lessons learned from previous 
projects. Management tracks costs, schedule, functionality, and problems. Management 
must initiate and champion the improvement effort. Effective customer-supplier 
relationships are also established with subcontractors. Only with management discipline 
can good software engineering practices be retained, especially during periods of crisis 
and pressure. The goal of Level 2 organizations is to institutionalize management 
practices shown to be successful in past projects, so that the same success can be repeated 
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in the future. SEI summarizes the process capability of Level 2 organizations as 
"disciplined" because project planning and tracking are stable and earlier successes can 
be repeated. [Ref. 7:p. 22] This level contains six KPAs which are: 

• Requirements Management - The purpose is to establish a common understanding 
of the project requirements between the customer and the development team. This 
agreement forms the basis for estimating, planning, performing, and tracking the project’s 
software activities. Goals of this KPA are that system requirements allocated to software 
are controlled to establish a baseline for software engineering and management use. 
Software plans, products, and activities are kept consistent with the system requirements 
allocated to software. 

[Ref. 8:pp. 55-57] 

• Software Project Planning - The purpose is to establish reasonable plans for 
performing the software engineering and for managing the software project. It involves 
developing estimates for the work to be performed, establishing the necessary 
commitments, and defining the plan to perform the work. This plan is necessary for 
initiating the software effort and managing the work. Its goals include that software 
estimates are documented for use in planning and tracking the software project, software 
project activities and commitments that are planned and documented. It is also used for 
the affected groups and individuals that agree to their commitments related to the software 
project. [Ref. 8:pp. 58-62] 

• Software Project Tracking and Oversight - The purpose is to provide adequate 
visibility of the actual progress being made so that management can take corrective 
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actions when performance deviates significantly from the plan. It involves tracking and 
comparing software accomplishments and results against documented estimates and plans. 
The plans and estimates are adjusted as necessary. The goals of this KPA are that actual 
results and performances are tracked against the software plans. Corrective actions are 
taken and managed to closure when actual results and performance deviate significantly 
from the software plans. Changes to software commitments are agreed to by the affected 
groups and individuals. [Ref. 8:pp. 63-64] 

• Software Subcontract Management - It provides a means to select qualified 
software subcontractors and manage them effectively. This involves selecting a software 
subcontractor, establishing commitments with the subcontractor, and tracking and 
reviewing the subcontractor’s performance and results. "Qualified" does not mean a 
subcontractor with the "highest process capability." The intent is to find one that is 
capable of meeting the requirements. This KPA’s goals are that the prime contractor 
selects qualified software subcontractors, the prime contractor and the software 
subcontractor agree to their commitments to each other, the prime contractor and the 
software subcontractor maintain ongoing communications, and the prime contractor tracks 
the software subcontractor’s actual results and performance against its commitments. [Ref. 
8:pp. 65-67] 

• Software Quality Assurance - Its purpose is to provide management with 
appropriate visibility into the process being used and the products being built. This 
requires reviewing and auditing the software products and activities to verify they comply 
with the established procedures and standards. The development team and management 
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axe provided with the results of those reviews and audits. Software quality assurance 
personnel should be independent of the software producers and project managers. The 
goals include that software quality assurance activities are planned; software products and 
activities adhere to applicable standards, procedures, and requirements and are verified 
objectively; affected groups and individuals are informed of software quality assurance 
activities and results; and that noncompliance issues that cannot be resolved within the 
software project are addressed by senior management. [Ref. 8:pp. 68-70] 

• Software Configuration Management - Its purpose is to establish and maintain the 
integrity of the products of the software project throughout the software life cycle. It 
involves identifying configuration items/units, systematically controlling changes, and 
maintaining the integrity and traceability of the configuration throughout the software life 
cycle. The goals are that software configuration management activities are planned; 
selected software work products are identified, controlled, and available; changes to 
identified software work products are controlled; and affected groups and individuals are 
informed of the status and content of software baselines. [Ref. 8:pp. 71-74] 

3. Level 3: Defined 

At the defined level, the standard process for both management and software 
engineering activities is documented, standardized, and integrated into what the CMM 
refers to as the organization’s standard software process. It includes inputs, standards, 
and procedures for performing the work, verification mechanisms such as peer reviews, 
outputs, and completion criteria. Because the process is well-defined, management has 
good visibility on the progress of all projects. An organization-wide training program is 
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established to ensure all personnel possess the knowledge and skill required to implement 
the standard software process. They are also taught the roles and responsibilities of 
management and the development team during each phase of the process. Project teams 
use an approved, tailored version of the organization’s standard software process for 
developing and maintaining software. It takes into account the project’s unique 
characteristics. [Ref. 7:p. 22] 

The software process capability of a Level 3 organization is summarized as standard 
and consistent by the SEI. This is because both software engineering and management 
activities are stable and repeatable. Cost, schedule, and functionality are under control 
and quality is tracked. Process capability is based on a common understanding of the 
activities, roles, and responsibilities in a defined process throughout the organization. 
[Ref. 7:p. 22] The defined level contains seven KPAs, which include: 

• Organization Process Focus - Its purpose is to establish the organizational 
responsibility for software process activities that improve the organization’s overall 
process capability. It involves developing and maintaining an understanding of the 
organization standard software process and the tailored project software process. Efforts 
are also directed at coordinating the activities to assess, develop, maintain, and improve 
these processes. The typical mechanism to accomplish this is the Software Engineering 
Process Group (SEPG). Other mechanisms include process review boards, quality circles, 
and process steering committees. The goals of this KPA are: the software process 
development and improvement activities are coordinated across the organization, the 
strengths and weaknesses of the software process used are identified relative to a process 
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standard, and the organization level process development and improvement activities are 
planned. [Ref. 8:pp. 79-81] 

• Organization Process Definition - This KPA aims to develop and maintain a set 
of software process assets that improve process performance and provide a basis for 
cumulative, long term benefits. Software process assets include the organization’s 
standard software process, descriptions of the software life cycles approved for use, the 
guidelines and criteria for tailoring the organization’s standard software process to fit the 
project’s uniqueness, the organization’s software process database, and the library of 
software process related documentation. It involves developing and maintaining the 
organization’s standard software process and related process assets. Its two goals are: 
standard software process for the organization is developed and maintained, and 
information related to the use of the organization’s standard software process by the 
software projects is collected, reviewed, and made available. [Ref. 8:pp. 82-89] 

• Training Program - The objective is to develop the knowledge and skills of 
individuals so they can perform their required roles effectively and efficiently. It entails 
identifying the training needs of the organization and project teams. It also includes 
developing and/or procuring training to address these needs. Training may include 
informal as well as formal methods. Training at this level differs from that at Level 2. 
Unlike Level 3, training at Level 2 is not likely to have been institutionalized across the 
organization. The goals are that training activities are planned, training for developing 
the skills and knowledge needed to perform software management and technical roles is 
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provided, and individuals in the software engineering group and other software related 
groups receive the training necessary to perform their roles. [Ref. 8:pp. 90-92] 

• Integrated Software Management - It is used to integrate the project’s software 
engineering and management activities into a coherent, defined software process tailored 
from the organization’s software process assets. It involves developing the project’s 
defined software process by tailoring the organization’s standard software process and 
managing it accordingly. The goals of Integrated Software Management are that the 
project’s defined software process is a tailored version of the organization’s standard 
software process and the project is planned and managed according to the project’s 
defined software process. [Ref. 8:pp. 93-94] 

• Software Product Engineering - Its purpose is to consistently perform a well- 
defined engineering process that integrates all the software engineering activities to 
produce correct, consistent software products effectively and efficiently. This requires 
performing the engineering tasks such as requirements analysis, design, code, and test, 
and to build and maintain software using appropriate tools and methods. The goals of 
this KPA are that software engineering tasks are defined, integrated, and consistently 
performed to produce the software, and secondly, that software products are kept 
consistent with each other. [Ref. 8:pp. 95-97] 

• Intergroup Coordination - This is used to establish a vehicle for the software 
engineering group to participate actively with the other groups so the project is better able 
to satisfy the customer’s requirements. These other groups may exist within or outside 
the organization. Examples include systems engineering, marketing, and training. It 
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involves the disciplined interaction and coordination of the project engineering groups 
with others to address system level requirements, objectives, and plans. Its goals include 
that the customer’s requirements are agreed to by all affected groups, the commitments 
between the engineering groups are agreed to by the affected groups, and that the 
engineering groups identify, track, and resolve intergroup issues. [Ref. 8:pp. 98-100] 

• Peer Reviews - Such reviews aim to remove defects from the software products 
early and efficiently. An important corollary is to develop a better understanding of the 
software process and potential defects that can be prevented. It requires a methodical 
examination of work products by the producer’s peers to identify defects and areas where 
changes are needed. Possible alternative ways of implementing peer reviews include 
Fagan-style inspections, structured walkthroughs, and active reviews. The goals are that 
peer review activities are planned and that defects in the software work products are 
identified and removed. [Ref. 8:pp. 101-103] 

4. Level 4: Managed 

At the managed level, the software development is well-defined and the organization 
sets quantitative quality goals for both products and processes. Both the software process 
and products are quantitatively understood and controlled. The principles of statistical 
process control are used to measure the software process and product quality for 
important activities across all projects. The results are stored in an organizational 
database, which is used to collect and analyze the data available from a project’s defined 
processes. These measurements are used in evaluating a project’s processes and products. 
Projects control their products and processes by narrowing the variation in performance 
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to within acceptable quantitative ranges. Sources of variations are discovered and 
corrected. [Ref. 7:p. 22] 

SEI summarizes the software process capability of Level 4 organizations as being 
quantifiable and predictable because the process is measured and operates within 
acceptable measurable limits. This level of capability lets an organization predict trends 
in process and product quality. Because the process is both stable and measured, when 
some exceptional circumstance occurs the organization is able to identify and address 
their causes. When the measurements exceed acceptable ranges, managers take action to 
correct the situation. [Ref. 7:p. 22] There are two KPAs for this level: 

♦ Quantitative Process Management - The KPA quantitatively controls the process 
performance of the software project. It involves establishing goals for process 
performance, measuring the performance of the project, analyzing these measurements, 
and making adjustments to maintain process performance within acceptable limits. Its 
goals are that the quantitative process management activities are planned, the process 
performance of the project’s defined software process is controlled quantitatively, and the 
process capability of the organization’s standard software process is known in quantitative 
terms. [Ref. 8:pp. 108-112] 

♦ Software Quality Management - The goal is to develop a quantitative measurement 
of the quality of the project’s software products and specify and achieve quality goals. 
It requires defining quality goals for the software products, establishing plans to achieve 
these goals, monitoring and adjusting software work products and development activities 
to meet quality goals to satisfy the needs of the customer. The goals for this KPA are 
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that the project’s software quality management activities are planned; measurable goals 
for software product quality and their priorities are defined; and that actual progress 
toward achieving the quality goals for the software products is quantified and managed. 
[Ref. 8:pp. 113-114] 

5. Level 5: Optimizing 

At the optimization level, the entire organization is focused on continuous process 
improvement. Continuous process improvement is based on quantitative feedback. 
Project teams analyze defects to determine their cause, evaluate the process to prevent 
known types of defects from reoccurring, and disseminate lessons learned to other 
projects. Reducing waste happens at all maturity levels, but is the focus of Level 5. 
Waste is unacceptable, and organized efforts are directed at removing waste. Data on 
process effectiveness are used to perform cost benefit-analysis of new technologies and 
propose changes to the process. Innovations that exploit the best software engineering 
practices are identified and used throughout the organization. The software process is 
continuously improved in a controlled manner. [Ref. 7:p. 23] 

SEI summarizes the software process capability of Level 5 organizations as 
"continuously improving." Because these organizations continuously strive to improve 
their process capability, they can expect to improve the quality of their products. 
Improvement occurs both by incremental advancements in the existing process and by 
innovations in technologies and methods. Technology and process improvements are 
planned and managed as part of the normal process of doing business. Level 5 is not the 
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final destination but the foundation for building an ever improving process capability. 
[Ref. 7:p. 23] The Level 5 KPAs are as follows: 

• Defect Prevention - Its purpose is to identify the cause of defects and prevent 
them from reoccurring. This involves analyzing defects that were encountered in the past, 
and taking specific actions to prevent their reoccurrence in the future. Its goals are that 
defect prevention activities are planned, common causes of defects are sought out and 
identified, and common causes of defects are prioritized and systematically eliminated. 
[Ref. 8:pp. 118-120] 

• Technology Change Management - The KPA identifies new technologies and 
incorporates them in the organization’s standard software process in an orderly manner. 
New technologies include tools, methods, and processes. The goals for this KPA are that 
incorporation of technology changes is planned, new technologies are evaluated to 
determine their effect on quality and productivity, and appropriate new technologies are 
transferred into normal practice across the organization. [Ref. 8:pp. 121-123] 

• Process Change Management - Its aim is to continuously improve the software 
process with the intent of improving software quality, increasing productivity, and 
decreasing the cycle time for product development. This involves defining process 
improvement goals, and systematically identifying, evaluating, and implementing 
improvements to the organization’s standard software process and the project’s defined 
software processes. The goals are that continuous process improvement is planned, 
participation in the organization’s software process improvement activities is organization- 
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wide, and that the organization’s standard software process and the projects defined 
software processes are improved continuously. [Ref. 8:pp. 124-125] 

D. PLANNED IMPROVEMENTS 

During the next few years, the current version CMM will continue to undergo 
extensive testing through the use of SPAs, SCEs, and process improvement programs. 
Feedback from these activities, Government, and the software industry will be used to 
improve the CMM. Version 1.1 is expected to be in use until at least 1996. The release 
of CMM version 2.0 is planned for the 1996-1998 time frame. This provides a balance 
between the need for stability and the goal of continuous improvement. [Ref. 8:p. 27] 

Version 2.0 will incorporate suggested improvements by the user community. SEI 
is also working with the International Standards Organization (ISO) in an effort to build 
international standards into the CMM, SPA, SCE, and other process improvement 
activities. [Ref. 7:p. 27] While all levels of the model may be revised, the focus will be 
directed at Levels 4 and 5 [Ref. 8:p. 141]. SEI believes that the KPAs for Levels 2 and 
3 have been almost completely defined. Since few organizations have been assessed to 
be at Levels 4 or 5, little is known about the characteristics of such an organization. The 
KPAs of Levels 4 and 5 will be refined as SEI works closely with organizations striving 
to understand and achieve these levels. The CMM may also expand its scope to address 
technology and human resource issues. [Ref. 9:p. 5.3] 
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E. SUMMARY 


This chapter introduced the reader to the CMM. The SEI developed the CMM to 
assist software development organizations in increasing their process capability. The SEI 
is constantly improving the CMM based on feedback from the CMM user community. 

The next chapter will provide an overview of the SCE, which is one of the CMM- 
based assessment tools created by the SEI. 
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m. THE SOFTWARE CAPABILITY EVALUATION 


The SEI’s Software Capability Evaluation (SCE) is a method for evaluating the 
software development process of an organization [Ref. 5:p. 15]. Its primary purpose is 
to support the acquisition of software by assessing an offeror’s software process 
capability. A basic tenet of the SCE Method and CMM is that the more mature the 
contractor’s software process capability is, the more likely it is that the contractor can 
meet cost, schedule, and performance targets. 

The SCE provides a snapshot of a contractor’s past process implementation, current 
process activities, and future process potential. Four to six Government personnel 
conduct a three day in-plant review at an offeror’s facility. The results of the SCE are 
the strengths, weaknesses, and improvement activities measured against the CMM. 
[Ref. 10] 

Use of the SCE method within DOD indicates that SCEs provide the following 
benefits for PMs: 

♦ Adds Software Development Capability Realism to the Source Selection Process - 
The PM can compare SCE results with the evaluation of the contractor’s proposal and 
software development plan (SDP) to determine whether the proposed approach is realistic 
in light of the offeror’s current process capability. 
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♦ Increases Objectivity In Information Collected For an Acquisition - The SCE 
method helps ensure an objective review by putting a trained evaluation team on site to 
evaluate the offeror’s activities and compare them against a public standard, the CMM. 
Each offeror is evaluated using the same criteria. 

♦ Motivates the Contractor’s Software Process Improvement Actions - By malting 
SCE results a discriminator in source selection, contractors should be motivated to focus 
on software process improvement in order to retain or increase Government business. 
[Ref. 10:pg. 3-52 - 3-53] 

A. DEVELOPMENT 

At the request of the USAF, SEI and the MITRE Corporation conceived and 
developed the SCE Method in 1987. It was designed to help PMs determine the software 
process capability of a contractor at one organizational site (facility or location). [Ref. 9] 
The original version of the SCE was first described in the 1987 SEI publication A 
Method for Assessing the Software Engineering Capability of Contractors. Over several 
years, major changes to the original version have occurred based on feedback from SCE 
users. These changes include the elimination of maturity level scores, shifting from being 
a "question-based" to a "model-based" method, and public baselining of the SCE Method. 
[Ref. 5:p. 29] 

B. CRITICISMS AND CHANGES 

The SCE methodology and its use have not gone unchallenged. The result of the 
original SCE method provided to a Source Selection Evaluation Board (SSEB) was a 
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calculated maturity level for an organization’s software development process. A 
contractor was scored as a "Level 1" to "Level 5" organization. This was based on a 
formula that used the number of verified "yes" answers to the SEI’s maturity 
questionnaire by the contractor. [Ref. 5:p. 29] 

There were problems with expressing the SCE results as a single maturity level 
score. The single maturity level score did not provide the SSEB with the detailed 
information on a contractor’s software development process that it required to evaluate 
an offeror [Ref. 5:p. 29]. In addition, issues have been raised by the software industry 
on the statistical reliability of the formula used to calculate an organization’s maturity 
level: 

By breaking the test into a multihurdle structure, the statistical reliability of the answer 
set is reduced in two ways. First, each hurdle or minitest will be less reliable simply 
because it has fewer questions. Second, the way the test is linked into a chain means 
that the uncertainty of the individual minitests must be multiplied by each other. Taken 
together, these effects result in a very rapid escalation of statistical uncertainty as the 
total number of hurdles is increased. [Ref. 11]. 

More recently, there has been a concern with Government organizations specifying that 
a minimum maturity level be achieved in order to be considered responsive to a 
solicitation [Ref. 12][Ref. 13]. To correct some of these problems, the 
SCE version 1.5 results are expressed as strengths, weaknesses, and improvement 
activities at the KPA level [Ref. 5:p. 29]. 

Another major change from the original SCE version was to shift the focus of the 
evaluation from the maturity questionnaire to the CMM. The original goal of the SCE 
was to validate a vendor’s answers to the maturity questionnaire. There were two major 
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problems encountered when using the original SCE. First, not all KPAs were adequately 
covered by the questionnaire, and second, some questions were not based on KPAs at all. 
[Ref. 6:pp. 29-30] 

To resolve these problems, the SEI has shifted the focus of the SCE version 1.5 
from validating an offeror’s response to the maturity questionnaire towards examining the 
KPAs of the maturity levels of the CMM [Ref. 6:p. 30]. In addition, a SEI representative 
stated in an interview with the researcher that the SEI has just published a revised 
questionnaire correcting these problems. 

Another major criticism was that there was no public description of the SCE 
process. Detailed information about the SCE was only available through the training 
given to Government SCE teams. The secrecy of the method, especially considering the 
millions of dollars of potential Government contract work at stake, drew criticism from 
the software industry [Ref. ll:p. 28]. In July 1993, the SEI published CMU/SEI-93-TR- 
17 Software Capability Evaluation (SCE) Version 1.5 Method Description. The SCE 
method description publication establishes a baseline. This will permit input into the 
future evolution of the SCE by organizations other than the SEI and SCE users. 

C. SCE VERSION 1.5 

The SCE version 1.5 consists of five major activity phases: Evaluation Start, 
General Preparation, Specific Preparation, Site Data Collection, and Findings. The 
method is summarized in Figure 1. A brief discussion of each phase follows. 
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Phase 

EVALUATION 

START 


GENERAL 

PREPARATION 


SPECIFIC 

PREPARATION 


SITE DATA 
COLLECTION 


FINDINGS 


Major Activities and Outcome 

The PMO: 

* Determines the attributes of the desired software product and 
the project required to produce it, 

* Determines the process capability that is most appropriate for 
the planned development (Target Process Capability) 

* Selects the SCE team 

Outcome: Decision to use the SCE Method 

The SCE Team: 

* Identifies areas where the development organizations lack 
experience (indicating potential risk) 

* Defines the scope of the SCE down to the level of subprocess 
areas that will be investigated at all of the development 
organizations 

Outcome: Scope of SCE defined and hitfi level preparations for 
evaluating all development organizations completed 

The SCE Team: 

* Selects projects for evaluation 

* Prepares specific topics corresponding to the subprocess area 
for evaluation; topics address observable work practices 

* Coordinates preparation for the Site Data Collection activities 

Outcome: Detailed preparations for evaluating a particular 
development organization site completed 

The SCE Team: 

* Visits the site and investigates each subprocess area topic 

* Determines strengths, weakness, and/or improvement activities 
through interviewing and document review 

Outcome: Processes at a particular site are investigated 


The SCE Team: 

* Consolidates the information collected on site by KPA in terms of 
strengths, weakness, and observed improvement activities 

Outcome: Findings of the investigation are documented 


Source: CMU/SEI-93-TR-17 


Figure 1 SCE Phases and Major Activities 
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1. Phase 1: Evaluation Start 

In this phase, the Program Management Office (PMO) decides to use the SCE 
method and begins the required planning and preparation. The purpose of this phase is 
to determine the role of the SCE, define the attributes of the desired software product, 
determine the process capability required for the acquisition, and select the SCE team. 
This phase is performed by the PMO. Subsequent phases are conducted by the SCE 
team. [Ref. 5:pp. 20-21] 

The first step is to decide whether to use the SCE. The SCE method was not 
developed to be used for all software acquisitions. PMs should consider the costs and 
benefits of using the SCE, as well as the unique requirements of their program, when 
making this decision. For small acquisitions, the costs may exceed the expected benefits 
of conducting an SCE. Figure 2 provides an application guide to assist the PM in 
deciding whether to use the SCE. [Ref. 10:pp. 3-45 - 3-46] 

For all developments that involve significant Mission Critical Computer Resources 
(MCCR) however, the SEI strongly recommends using the SCE for source selection. The 
success of such systems is heavily dependent on its embedded software. The process 
capability of prospective vendors must be evaluated to select the contractor with the 
highest probability of success. [Ref. 10:p. 3-47] 

Once the decision is made to use the SCE, the PM begins planning the activities 
required to incorporate the SCE into the source selection process. Details to be addressed 
include: determining the weight of the results of the SCE; including the intention to use 
the SCE in the Commerce Business Daily (CBD) announcement, Acquisition Strategy, 
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Request For Proposal (RFP), and other important acquisition documents; scheduling of 
SCE activities; and identifying and allocating resources in terms of dollars, time, and 
personnel to support the evaluation. [Ref. 5:p. 21] The SEI has provided examples of the 
recommended wording to use for SCE input to the CBD and RFP [Ref. 10:pp. 5-87 - 5- 
111]. The SEI also provides a sample schedule of SCE activities [Ref. 10:pp. 4-64, 4-67]. 

To conduct a SCE, the PMO must first define the software development project by 
creating a target product profile. The target product profile represents the "customer’s 
view" of the product to be built [Ref. 5;p. 150]. The target product profile defines the 
proposed software project in terms of major and minor attributes. Major attributes include 
the application domain, product type, size, type of work, use of subcontractors, and 
previous vendor experience in developing this product type. Minor attributes, for which 
estimates are made, include the programming language required, target hardware 
configuration, applicable developmental standards, the customer, host development 
system, and configuration management tools. [Ref. 5:pp. 171-174] 

Once the target product profile has been created, the next step is to determine the 
target process capability - the process capability that is most appropriate to the planned 
development [Ref. 5:p. 17]. Senior software engineers within or assisting the PMO 
determine which KPAs are required to successfully develop the proposed software system. 
These KPAs are matched to the CMM to determine the highest maturity level required 
by a prospective vendor. This maturity level becomes the target process capability. [Ref. 
5:pp. 43-45] 
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Based on the CMM principle that higher levels of maturity cannot occur without 
first satisfying all the KPAs of the lower levels, all KPAs from Maturity Level 2 to the 
target process capability should be included in the SCE, whether they were determined 
to be important to the software project or not. At a minimum, the KPAs of Maturity 
Level 2 (Repeatable) should be evaluated. [Ref. 6:p. 45] 

During this phase, the PMO requests important information from the vendors that 
will be used in later phases. A proposed project profile depicts the "vendor’s view" of 
the proposed system. It is similar in form to the target product profile. Six to eight 
project profiles (also in the form of the target product profile) of ongoing or completed 
projects are submitted by the vendor as candidates for evaluation during the site visit. 
These projects should be similar to the proposed project profile submitted by the vendor. 
Organization charts are requested for the entire organization and for the development 
teams of the candidate projects. Answers to the questionnaire by the organization as a 
whole and the project teams are also requested at this time. The request for information 
is usually made in the Request for Proposal (RFP). [Ref. 5:p. 53] 

The last step in this phase is selecting the team members. A SCE team consists of 
four to six personnel trained in the use of the SCE. The team should average a minimum 
of seven years of software development or management. Experience in software 
acquisition is also helpful. At least two members of the team should have previous 
experience in conducting SCEs. No more than one member should have less than two 
years of professional software experience. Collectively the team must have knowledge 
and experience with the following: 
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• The application domain and product type. 

• The management processes required to create an effective environment for the 
engineering and development of a software product. 

• The major phases that engineering and development of a software product must 
go through. 

• The support processes and management environment required to reduce or 
eliminate unnecessary rework within the engineering and development of a software 
product. 

• The relationship between technology (in the form of methods and tools) and 
support processes. [Ref. 5:pp. 46-47] 

2. Phase 2: General Preparation 

In this phase, the SCE team completes the high-level preparations for evaluating all 
offerors. In the first phase, the scope of the SCE was defined to the KPA level. The 
purpose of this phase is to narrow the scope to critical subprocess areas of all the KPAs 
within the target process capability. [Ref. 5:p. 49] 

To determine the critical subprocess areas, the SCE team compares the proposed 
project profile for an organization with its project profiles of the projects submitted as 
candidates for evaluation. The team looks for mismatched attributes. A mismatched 
attribute exists if an organization’s project profiles do not match any of the proposed 
project profiles. The target project profile and the proposed project profile are also 
compared. Any significant differences in major attributes can also be treated as a 
mismatch. [Ref. 5:pp. 52-55] The SEI has developed tables that identify subprocess areas 
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that should be evaluated if mismatches in major attributes occur. The SEI has also 
identified subprocess areas that should be used for all SCEs. [Ref. 6:pp. 175-183] These 
subprocess areas identified for evaluation are combined into one list. The list is tailored 
by eliminating subprocess areas that are not part of targeted KPAs, and including at least 
one subprocess area for each target KPA. Additional subprocess areas are added within 
the targeted KPAs based on the judgment of the SCE team. The tailored list defines the 
critical subprocess areas that are evaluated during the SCE for each offeror. [Ref. 5:pp. 
56-59] 

3. Phase 3: Specific Preparation 

In this phase, the SCE team conducts detailed preparations for conducting a three- 
day site visit for each offeror. During the general preparation phase, planning focuses on 
issues that affect all the site visits. In this phase, the SCE team focuses on preparing for 
each individual site visit. The high-level preparations accomplished during the general 
preparation phase are refined and tailored for each offeror. The purpose of this phase is 
to prepare the SCE team for a specific site visit. [Ref. 5:p. 24-25,62] 

Site visits are only conducted on those offerors within the competitive range [Ref. 
10:p. 4-66]. A minimum maturity level should never be used as a screening factor in 
determining contractors within the competitive range [Ref. 10:p. 5-89]. 

The first step in this phase is to select the projects that will be evaluated for each 
offeror. The SCE team’s goal is to evaluate risk in the processes that the offeror is 
planning to use on the project. It does this by comparing the project profiles and 
proposed project profile submitted by an offeror. It then selects three to four projects that 
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closely resembles the proposed project profile in terms of the major attributes that were 
described earlier in this thesis. By examining the processes used in these projects, the 
team hopes to gain a clear understanding of the processes that will probably be used on 
the subject acquisition. [Ref. 5:pp. 64-66] 

Once the projects are selected, the SCE team requests documentation for review. 
This typically includes policies, standards, procedures, and directives relating to software 
development both at the organizational level and from the development teams for the 
projects that have been selected for evaluation. [Ref. 5:p. 65] 

The SCE team also creates a key issue worksheet for each offeror. All critical 
subprocess areas and mismatches identified during the general preparation phase are listed 
on the key issue worksheet. The team then takes the offeror’s answers to the 
questionnaire and looks for inconsistencies and anomalies. An inconsistency is a 
contradictory response from the same project to two or more questions that relate to the 
same subprocess area. An anomaly is a contradictory response to the same question by 
two or more projects. Inconsistencies and anomalies are also recorded on the key issue 
worksheet. [Ref. 5:pp. 66-69] 

Using the key issue worksheet, the SCE team develops a list of topics to be 
investigated for each offeror. Topics are generated on subprocess areas that have 
mismatches, inconsistencies, or anomalies. Topics address the policies; roles and 
responsibilities; non-human resources; procedures and standards; training; adherence to 
policies, standards and procedures; and improvement activities concerning these 
subprocess areas. [Ref. 5:pp. 69-73] 
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Due to time constraints, all topics cannot be investigated. The SCE team must use 
its experience to balance the need for adequate coverage of critical subprocess areas 
against the limited time allotted for the site visit. It must limit the topics to those that 
will yield the most critical information. 

Once the list of topics has been finalized for each offeror, the SCE team begins 
planning the procedures for conducting the interviews. This is done in several steps. 
First, the team decides how much time will be allocated for each topic for the site visit. 
Next, the SCE team selects who within the offeror’s organization should be interviewed. 
Selection is done by position, not by name. More than one person can be interviewed for 
a topic and a person can be asked about several topics. Questions to be asked are 
finalized and documented. Finally, the team coordinates the interview schedule with the 
offeror to ensure that the interviewees are available at the designated times. It also 
coordinates the use of adequate facilities and administrative support at the vendor’s plant 
during the site visit. [Ref. 5:pp. 73-76] 

In the last step of this phase, the SCE team develops an initial briefing that will be 
given to all vendors. The agenda should include introducing the team members, 
describing the major on-site activities, discussing how the interviews will be conducted, 
and how the findings will be presented to the offeror. The initial meeting is 
recommended to last no more than 60 to 90 minutes. [Ref. 5:pp. 76-77] 

4. Phase 4: Site Data Collection (Site Visit) 

The Site Data Collection phase is the most important phase of the SCE method. 
It is during this phase that the SCE team evaluates the processes at a vendor’s site. 


38 



The purpose of this phase is to investigate the topics associated with each critical 
subprocess area in sufficient depth to determine strengths, weaknesses, and improvement 
activities for the corresponding subprocess area. [Ref. 5:p. 79] 

After conducting the initial briefing developed in the previous phase, the team 
begins collecting data. Site data collection has two basic components: investigation and 
decision making about the information collected. The team uses two complimentary 
means to investigate a topic: document reviews and interviewing. [Ref. 5:p. 79] 

The SCE method assumes that a process not documented is usually not followed. 
Documents define and standardize processes, indicate commitment to use the process, 
provide an audit trail of processes used, and collect data about process performance. The 
documents requested earlier are reviewed for information. Additional documents may be 
requested to provide more information, clarify issues, or show evidence of use. The time 
required for documentation review varies at each site. [Ref. 5:pp. 26,105-110] Examples 
of documents include organization charts, corporate policy/procedures on software 
management, project policy/procedures on software management, software development 
plans, software configuration management plans, software quality assurance plans, training 
plans, current metrics, and minutes of meetings [Ref. ll:p. 5-96]. 

Interviews provide insight into how the processes are implemented in practice, and 
show the extent to which the processes are understood by the developmental staff. The 
SCE method assumes that if a process is not understood by the people implementing it, 
it usually is not followed. [Ref. 5:p. 27] The number of people interviewed during a site 
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visit can be as few as ten or as many as 30 depending on the agenda developed by the 
SCE team during the specific preparation phase [Ref. 14]. 

Interviews are conducted involving one member of the development organization 
and the entire SCE team. The SEI believes that an employee will speak more freely 
without his supervisor and peers present. Also, data collection is more effective with the 
entire team conducting the interview. Because the entire SCE team is involved, the 
interview environment may be intimidating. Steps must be taken to put the interviewee 
at ease. [Ref. 5:pp. 111-113] 

There are two types of interviews used during the site visit, exploratory and 
consolidation. The first round of interviews conducted by the SCE team are called 
exploratory interviews. These interviews are conducted in accordance with the published 
interview schedule given to the vendor. Exploratory interviews provide insight into how 
process areas are implemented and identify the documents where these processes are 
found. The length of an exploratory interview depends on the number of topics the 
interviewee is asked to address and the depth of the information required by the SCE 
team. [Ref. 5:pp. 85-87] 

After the exploratory interviews are concluded, the SCE team conducts 
consolidation interviews to corroborate and clarify information that confirms or negates 
findings. During consolidation interviews, the SCE team questions individuals who may 
be able to provide additional objective evidence required to finalize the team’s findings. 
This may include individuals questioned during the exploratory interviews. Consolidation 
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interviews usually focus on one or two questions aimed at resolving open issues. They 
are therefore generally shorter in duration than exploratory interviews. [Ref. 5:p. 93] 

After each interview and document review, the team conducts a caucus. The 
purpose of the caucus is to analyze, share, and consolidate the considerable amount of 
data gathered so far. Team members share their analysis with each other to form a group 
consensus on what the data indicate and whether enough information has been gathered 
to make conclusions on a topic. If more information is required, the SCE team decides 
what is needed and which method to use, document review or interview, to get it. [Ref. 
5:pp. 88-89] 

A visual summary of the data collection process is depicted in Figure 3. 

5. Phase 5: Findings 

This is the final phase of the SCE in which the team documents the results of its 
investigation. The purpose of this phase is to consolidate the decisions made during the 
site data collection phase into findings at the KPA level. Decisions made by the SCE 
team during the caucuses address specific topics. These topics cover the critical 
subprocess areas, which in turn, are part of the targeted KPAs. The team takes the 
information it has collected and extrapolates the results to the KPA level in the form of 
strengths, weaknesses, and improvement activities. [Ref. 5:pp. 99-101] 

The actual findings are normally completed at the vendor’s facility, although the 
final report on the findings may be done later. If permitted by the contracting officer, the 
SCE team briefs the offeror on the results before leaving. A final findings report is 
tailored and provided to each offeror. It includes information common to all site visits 
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Figure 3 Site Data Collection Activities 

such as the target product profile, target process capability, and critical subprocess areas. 
Information submitted by the offeror, including its proposed project profile, project 
profiles, organization charts, and other documentation, are also included. All worksheets, 
evidence used in forming the finding, and the actual findings are included in this report. 
The offeror is encouraged to incoiporate these finding into its process improvement 
activities. Once the final report on the findings are published, and submitted to the SSEB, 
the SCE is completed. [Ref. 5:pp. 101-102] 
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The SEI makes a clear distinction that SSEB and Source Selection Advisory Council 
(SSAC) activities that incorporate the results of the SCE as part of the source selection 
process are separate from the SCE method [Ref. 5:p. 97]. SEI recommends that the SCE 
team leader be a member or advisor to the SSEB to work with the SSEB members in 
applying the evaluation standards to the SCE findings of each offeror [Ref. 10:p. 5-110]. 
This duty is performed outside the SCE, however. 

D. SUMMARY 

This chapter provided the reader with an overview of the SCE method. The 
benefits of the SCE include gaining insight into an offeror’s software process capability 
and promoting process improvement within the software industry. In the next two 
chapters, the reader will be introduced to the SDCE and its model. 
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IV. THE SOFTWARE DEVELOPMENT CAPABILITY EVALUATION 
MODEL 

The SDCE method has two major components, a model that structures the SDCE 
criteria and questions, and a set of activities used in applying the model to a specific 
source selection [Ref. 2:p. 12]. This chapter begins with a description of the SDCE 
method, its purpose, and expected benefits and limitations. The remainder of the chapter 
provides an overview of the SDCE model. A more detailed description of the model 
criteria and questions are found in Chapter VI of AFMC Pamphlet 800-61 Acquisition 
Management Software Development Capability Evaluation Volume 1. The following 
chapter will discuss the SDCE activities for applying the SDCE model. 

A. THE SOFTWARE DEVELOPMENT CAPABILITY EVALUATION METHOD 

From several interviews with individuals involved with the development of the 
SDCE, the author learned that in August 1992, AFMC created a process action team 
(PAT). Its original purpose was to evaluate the SCE and SDCCR with the goal of 
adopting one method for use in AFMC source selections. The PAT’s findings concluded 
that neither method was completely suitable. The PAT was then tasked to develop a 
software capability evaluation for AFMC. The result was the SDCE. 

An ASC/AFMC representative stated in an interview that the SDCE method has 
been approved for pilot use on the Tri-Service Standoff Missile (TSSM) and three other 
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programs from the Space Systems Division. SDCE preparations are ongoing to support 
source selection activities for these programs which are scheduled for fiscal year 95. 
Pending the results of the SDCE’s trial use, it is AFMC’s goal to adopt the SDCE as its 
only software capability evaluation. 

SDCE is a method used during source selection to evaluate an offeror’s software 
engineering and management capabilities. It also evaluates an offeror’s systems 
engineering capabilities which directly impacts software development. [Ref. 3:p. 1] 

The primary purpose of the SDCE is to increase the probability of selecting an 
offeror capable of successfully developing software that meets all requirements within 
cost and schedule constraints. Its secondary purpose is to gain contractual commitment 
to implement the methods, tools, practices, and procedures necessary for success. [Ref. 
3:p. 2] 

The SDCE was designed to be used on all programs where software development 
is vital to the system. It is primarily applicable for use in source selection for the 
Engineering and Manufacturing Development Phase (EMD) contract. But it may also be 
used for the Demonstration and Validation Phase (DEM/VAL) and for major system 
modifications and/or upgrades contracts. It is intended to be applied to the prime 
contractor and its associate contractors and subcontractors involved in the planning and 
development of software. [Ref. 3:p. 4] 

Although the SDCE method assists the SSA in selecting a capable contractor, the 
SDCE has limitations as well as benefits. The benefits include: 
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• The offeror is required to provide a comprehensive description of the software 
development capabilities in terms of engineering and management processes, methods, 
tools, and resources it proposes to use on the project. 

• The SDCE method reviews the systems engineering and other development 
disciplines and processes that are directly related to software development. 

• The SDCE method seeks to gain contractor commitment to follow well-defined 
processes described in the software development plan and are tied directly to the systems 
engineering master schedule. 

• It promotes team building between the Government and the contractor by 
developing a mutual understanding of the offeror’s capability and processes during the 
site visit. 

• It also reduces program risk by focusing on the weaknesses in a contractor’s 
software capability and process early in the development of the program. 

The limitations of the SDCE method include: 

• It does not develop program requirements found in the statement of work, 
specifications, and the contract data requirements list. 

• Although the SDCE reviews an offeror’s methods for estimating and generating 
software development schedules, it does not ensure an achievable software development 
schedule will be established and followed. 

• It does not specify a software design solution to the program requirements. 

• It cannot ensure that the resources that were evaluated during the SDCE will in 
fact be applied by the contractor to the program after contract award. [Ref. 3:pp. 4-5] 
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B. THE SOFTWARE DEVELOPMENT CAPABILITY EVALUATION MODEL 


The SDCE model contains critical capabilities that have historically been the high- 
risk areas in the development of software-intensive systems. The model organizes these 
critical capabilities into a three-level structure to facilitate the use of the SDCE method 
during source selection. The model is important because it contains the criteria and 
questions used by the SDCE team in evaluating an offeror’s software development 
capability. 

C. MODEL DEVELOPMENT 

The SDCE model was developed from many sources. It is primarily based on 
elements of the CMM, SPA, SCE, and the SDCCR. These models and methods have 
been used extensively in the USAF for acquisitions and process improvements. They 
have all been subjected to extensive reviews. While they formed the baseline for the 
SDCE model during development, the model corrects some of the shortfalls that were 
previously identified in feedback from the software community. For example, the CMM 
does not address systems engineering issues in software development, or human resources. 
The SDCCR did not evaluate process improvement efforts, defect prevention, metrics, and 
technology assessment and transition. [Ref. 3:p. 12] 

As mentioned previously, AFMC created a PAT to develop the SDCE method. The 
PAT consisted of personnel from AFMC and the SEI to address Government 
requirements, and industry representatives to address their needs and provide industry 
input into the development of the SDCE method. The PAT was composed of two teams. 
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One team developed the SDCE model, while the other team created the SDCE activities 
for applying the model. [Ref. 15] 

D. MODEL STRUCTURE 

The SDCE model is composed of three parts: the model structure, a set of model 
criteria, and questions. The model structure is a hierarchical decomposition of the 
capabilities deemed critical in the source selection process for software intensive systems. 
The structure facilitates the consolidation of SDCE findings in terms of strengths, 
weaknesses, and risk to the model structure level required by the source selection 
standards. The model structure has three levels: 

• Critical Capability (CC) - This is the lowest level of the model structure. A CC 
is a set of model criteria that, when evaluated together, provide the basis for identifying 
strengths, weaknesses, and risks. [Ref. 3:p. 260] 

• Critical Capability Area (CCA) - A CCA is a set of CCs that constitute an 
integrated development capability. The CCA facilitates the consolidation of strengths, 
weaknesses, and risks that were identified at the CC level. [Ref. 3:p. 260] • Functional 
Areas (FAs) - This is the highest level of the model structure. An FA is a set of related 
CCAs functionally grouped into major development capability areas. FAs are used to 
consolidate the strengths, weaknesses, and risks identified at the CCA level. [Ref. 3:p. 
260] 

The remaining components of the SDCE model are the model criteria and the 
questions. Model criteria are the defined standards for evaluating an offeror’s capability 
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Figure 4 The SDCE Model Structure 


and capacity in a standard and repeatable way. Model criteria address processes, people, 
tools, and technology. The questions part of the model consists of questions that are used 
to elicit information for specific model criteria. The questions ask an offeror to describe 
how it meets the model criteria. [Ref. 3:p. 14] Figure 4 depicts the model structure. 
Figure 5 provides an example of the SDCE criteria and questions. 
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Figure 5 Example of Model Criteria and Questions 
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E. SOFTWARE DEVELOPMENT CAPABILITY EVALUATION MODEL 

FUNCTIONAL AREA OVERVIEW 

There are six interrelated FAs for the SDCE model. Three of the FAs are layered 
on top of each other. Software engineering is part of systems engineering. It, in turn, 
is part of program management. The remaining three FAs cut across the first three FAs, 
in various degrees. For example, human resources is a part of the organizational resources 
and program support FA, yet it affects program management, software engineering, and 
systems engineering FAs. This is illustrated in Figure 6. [Ref. 3:p. 15] 

In the SDCE model, CCAs and CCs are assigned to only one FA even though they 
may be applicable to multiple FAs. CCAs and CCs are assigned to an FA based on a 
"best fit" determination based on the decisions of the SDCE PAT. [Ref. 3:p. 15] 

The SDCE model is depicted in Figure 7. The remainder of this chapter provides 
an overview of the FAs and CCAs of the SDCE model. 

1. Functional Area 1: Program Management 

The program management FA focuses on program management capabilities required 
to successfully develop and manage software. Its purpose is to evaluate whether program 
level management processes and procedures are established, and if they support the 
software engineering processes and procedures being used. The program management FA 
should also accurately track the cost, schedule, and technical progress of the program. 
[Ref. 3:pp. 15-16] This FA contains the following CCAs: 

• Management Authority, Responsibility, and Accountability - This CCA focuses 
on the organization structure and control processes. It evaluates the assignment of 
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Figure 6 SDCE Functional Areas 


responsibilities, span of control, and the interrelationship between software engineering, 
program management, and systems engineering. 

• Program Planning and Tracking - This CCA evaluates the processes used in 
program planning, developing the contract work breakdown structure (WBS), defining 
work packages, and defining the program schedule. The correlation between these 
planning and tracking processes is also evaluated. 

• Subcontractor Management - This CCA evaluates an offeror’s processes to control, 
maintain status, and report subcontractor development efforts. In particular, it looks at 
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the flowdown of development requirements through the Systems Engineering Management 
Plan (SEMP), Systems Engineering Master Schedule (SEMS), and Systems Engineering 
Detailed Schedule (SEDS) to the subcontractor. It also includes the review, test, 
integration, and software development planning of the subcontractor. The integration of 
subcontractor activities with the prime is also evaluated. 

• Legal and Contracting Issues - This CCA evaluates the offeror’s process for 
identifying proprietary and restricted rights software. It also looks at the procedures used 
to develop and support the software under restricted rights constraints. 

• Risk Control - This CCA evaluates an offeror’s process for identifying and 
managing program risks. [Ref. 3:pp. 26-28] 

2. Functional Area 2: Systems Engineering 

This FA focuses on those areas of systems engineering that have the greatest 
potential impact on the successful development of software. [Ref. 3:p. 16] It contains 
the following CCAs: 

• System Requirements Development, Management, and Control - This CCA looks 
at the processes for developing and allocating system level requirements down to the 
software level. It also determines whether the resulting software requirements are 
adequate. The procedures for managing requirement changes, ensuring requirement 
traceability from the system to software level, and whether software issues are addressed 
during requirements development at the systems level are also addressed. 

• Computer System Architecture Design and Review Process -This CCA addresses 
processes used to define the system level architecture design that includes hardware and 
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software components. It also evaluates the effectiveness of the system architecture design 
reviews and the procedures used to analyze the impact of changes to the architecture. 

• Supportability - This CCA evaluates processes that addresses reliability and 
maintainability issues. These are of great concern to the support organization. 

• Intergroup Coordination - This CCA looks at the processes that coordinate issues 
across different development groups. It also assesses the coordination among developer, 
customers, users, and testers. 

• Systems Engineering Planning - This CCA evaluates the processes for defining 
systems engineering methods and the coordination with the software engineering methods 
to be used. It also looks at the adequacy of the SEMP, SEMS, and SEDS. 

• System Integration and Test - This CCA assesses the process for integrating and 
planning. It also looks at the adequacy of the tools and facilities to support testing. 

• Reuse - This CCA evaluates the processes used to exploit the advantages of reuse. 
It addresses the ability to reuse existing components, to develop common components, 
and to develop new components with increased reuse potential. [Ref. 3:pp. 28-30] 

3. Functional Area 3: Software Engineering 

This FA evaluates the capabilities that will be used to manage the engineering 
development of the software product. The capabilities include the ability to create an 
acceptable software development plan (SDP); estimate software size, cost, and schedule; 
define development methodologies; track and report against the SDP; and develop and 
control software requirements, design, coding, integration, and testing. [Ref. 3:pp. 16-18] 
This FA encompasses the following CCA: 
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• Software Development Planning - This CCA ensures that the resources required 
to meet all requirements are identified, budgeted, and allocated to the program. 

• Software Project Tracking and Reporting - This CCA evaluates the process that 
keeps program and engineering management informed on the status of each software 
component and the program development as a whole. It also ensures that corrective 
actions are initiated and completed when necessary. 

• Software Requirements Management - This CCA looks at the processes used to 
analyze, use, and maintain the software requirements after they have been baselined. 
(Recall that the initial development of the software requirements is covered in the systems 
engineering FA). 

• Software Design - This CCA evaluates the procedures used to develop, document, 
and maintain the software design. 

• Software Coding and Unit Testing - This CCA assesses the processes used to 
develop object code and to perform component or unit testing. 

• Software Integration and Test - This CCA evaluates the processes used to integrate 
the software components then test them as a unit. [Ref. 3:pp. 30-32] 

4. Functional Area 4: Quality Management and Product Control 

This FA evaluates the processes that ensure and maintain software quality 
throughout the program’s life cycle. Quality management entails defining, planning, 
implementing, and monitoring quality goals. Product control involves identifying the 
software configuration, systematically controlling changes to the configuration, developing 
documentation, and maintaining the integrity and traceability of the configuration 
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throughout the development life cycle. [Ref. 3:p. 18] This FA includes the following 
CCAs: 

• Software Quality Management - This CCA examines management processes that 
support quality assurance functions. This includes defining quality goals, establishing 
plans to achieve these goals, and monitoring and adjusting the process to satisfy the needs 
of the customer. 

• Software Quality Assurance (SQA) - The software quality assurance CCA looks 
at the vendor’s quality assurance organization. This includes processes that ensure 
compliance with program standards and quality goals, reporting quality findings, and the 
elevation of unresolved quality problems to management levels above the project team. 

• Defect Control - Defect control evaluates the processes that identify causes of 
defects and defect prevention. This includes thorough analysis of past defects and taking 
specific actions to prevent their reoccurrence in the future. 

• Metrics - This CCA evaluates an offeror’s processes to quantitatively assess the 
system and software development status and to consistently report metric results 
internally, to the prime and/or subcontractors, and to the customer. 

• Peer Reviews - Peer Reviews are planned, methodical examinations of the 
software products by the producer’s peers with the goal of early identification of defects 
and areas requiring change. This CCA examines the processes for conducting peer 
reviews. 

• Internal Independent Verification and Validation (I3V&V) - This CCA looks at the 
processes for conducting IIV&V on critical software elements. It also ensures that the 
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development schedule can accommodate all the activities required for functional, 
performance, and documentation verification and validation. 

• Software Configuration Management - This CCA evaluates the processes for 
establishing and maintaining the integrity of the software configuration. This includes 
processes for developing the software configuration, systematically controlling changes 
to the configuration, and maintaining the integrity and traceability of the configuration 
throughout the life cycle. 

• Documentation - This CCA examines the processes used to develop and review 
the documentation required for the development. The documents include software 
requirement documents, software design documents, operator and maintenance manuals, 
test plans, and test procedures. [Ref. 3:pp. 32-35] 

5. Functional Area 5: Organization Resources and Program Support 

The purpose of this FA is to evaluate an organization’s resources that are available 
and applied to the development program. It seeks to determine whether they are 
sufficient. [Ref. 3:p. 18] It includes the following CCAs: 

• Organizational Standards and Procedures - This CCA looks at the areas that 
develops and maintains the organization’s policies, standards, procedures, and other 
instruments that define the processes used. The organization should also update these 
items to reflect lessons learned. 

• Facilities - This CCA ensures the required facilities to perform the system and 
software development functions are identified, developed, and allocated to the program. 
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• Training - Training develops the individual skills and knowledge required for 
personnel to perform their roles effectively and efficiently. This CCA assess a 
contractor’s process for doing this. 

• Human Resources - It is often heard that personnel are the greatest resource for 
the project [Ref. 2:p. 4]. This CCA ensures that human resources are available in the 
required quantities and skills. It effectively and efficiently manages the allocation of 
personnel to various developmental activities and also addresses retaining personnel for 
the life of the program. 

• Technology Assessment and Transition - This CCA identifies and assesses new 
technologies that would be used on a project. It includes the tools, methods, and 
processes. When found to be beneficial to a project, the organization would transition 
them into use in an orderly manner. 

• Organization Process Management - This CCA focuses on process improvement. 
It has the goals of improving quality, increasing productivity, and decreasing development 
time. 

• System/Software Engineering Environment (S/SEE) - A S/SEE ensures the 
availability of integrated software development tools to support the different development 
and management functions. It should be consistent with the processes, methodologies, 
and languages used in development. [Ref. 3:pp. 35-37] 

6. Functional Area 6: Program Specific Technologies 

This FA addresses the technologies or application areas that are specific to a 
development program. Additional CCAs for unique technology or application areas 
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applicable to the program may be developed under this FA. [Ref. 3:pp. 18-19] This FA 
encompasses the following CCAs: 

• Artificial Intelligence (AI) - This CCA evaluates an offeror’s experience and 
expertise in applying AI tools and techniques to software development. 

• Safety Critical Digital Systems - This CCA examines the offeror’s capability and 
capacity to incorporate Safety Critical Digital Systems into the development. 

• Complex Hardware Development - This CCA evaluates the offeror’s processes and 
procedures for managing and developing complex custom integrated circuits. 

• Database Management - This CCA evaluates the offeror’s ability to develop large 
databases. [Ref. 3:pp. 37-39] 

F. SUMMARY 

ASC/AFMC and the SDCE PAT developed the SDCE as a method to evaluate the 
software development capability of an offeror during source selection evaluations. The 
SDCE model contains those critical capabilities that have historically been high-risk areas 
during software development. The activities that tailor and apply the SDCE model to a 
specific acquisition are discussed in the next chapter. 
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V. THE SOFTWARE DEVELOPMENT CAPABILITY EVALUATION 
ACTIVITIES 


The SDCE method is composed of 13 activities that encompass the six functional 
areas of the SDCE model. To distinguish them from the six functional areas, these 
activities are identified with letters A through M as depicted in Figure 8 [Ref. 3:p. 19], 
Each of these 13 activities are further broken down into key tasks. This chapter will 
discuss the main features of each of the activities. The reader is directed to Chapter V 
of the A EMC publication for a more detailed discussion of the SDCE activities. 

A. ACTIVITY A: DETERMINE APPLICABILITY 

The decision to use the SDCE method must be made as early as possible in the 
acquisition cycle. This activity focuses on making the program office aware of the 
SDCE, familiarizing program office personnel with the SDCE method, assisting them in 
determining whether the SDCE is applicable and beneficial to their program, and 
obtaining a commitment to use the SDCE. [Ref. 3:p. 41] 

Within each USAF Logistics and Product Center, SDCE center offices have been 
established to assist PMOs and SDCE teams in tailoring and applying the SDCE method. 
Local SDCE center offices and PMs have an equal responsibility to coordinate with each 
other as early as possible to begin SDCE planning. The local center SDCE office should 
be aware of all new programs that are initiated in order to advise PMs on the SDCE. 
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Figure 8 Overview of the SDCE Method 
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PMs should be aware of the existence of the SDCE method and its use, and should 
contact the local SDCE center office as soon as their program has been established. [Ref. 
3:p. 41] 

The first step for the PMO is to establish a working group to work with the local 
SDCE center office to determine the applicability of the SDCE to the subject project. 
This group should be composed of a PMO representative, the senior program/project 
engineer, the senior software engineer, and the procuring contracting officer. Ideally, 
these same individuals should be part of the SDCE team if the decision is made to use 
the method. [Ref. 3:p. 41] 

The SDCE method is intended to reduce software development risks. An SDCE 
application guide is depicted in Figure 9. It identifies the factors that should be 
considered when determining the applicability of the SDCE to a program. Although not 
shown on this chart, the acquisition strategy must also be reviewed to determine the 
impact of the SDCE. [Ref. 3:pp. 43-45] 

A PMO’s resources are not unlimited. The cost of the SDCE must also be a factor 
in determining whether to use it. An estimate of the cost of conducting a SDCE is 
provided in Figure 10. [Ref. 3:pp. 45-46] 

Once the PMO working group has determined the costs of conducting the SDCE 
and the benefits it expects from the evaluation, it briefs the PM on its findings and 
recommendations. If the PM decides to use a SDCE, key leaders involved with the 
acquisition, both outside and within the PMO, are briefed on the expected benefits and 
costs. Outside leaders include the Program Executive Officer (PEO), and directors from 
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The SDCE process should be applied during source selection 
on weapon systems entering the EMD phase when two or more 
of the following conditions, requirements, or 
characteristics exist. If only one exists, applying the 
SDCE may still be 

appropriate for particular acquisitions. 

• The development is designated a major program. 

• The system software is estimated to cost more that 
$25 million or require more than 100,000 SLOC. 

• The program involves highly complex requirements 
and an associated complex software development 
process. 

• A complex software to systems integration effort 
is expected. 

• The software development is constrained by an 
aggressive program schedule or it is anticipated 
that the software development will be on the 
project's critical path. 

• The program development involves safety critical 
software (e.g. human safety or nuclear surety 
factors). 

• The program development includes unprecedented 
functional capabilities or new software 
technologies. 

• It is anticipated that there may be bidders with 
uncertain software development and management 
capabilities, or unknown experience with the 
application domain. 

• The program is software intensive. 

The SDCE process should be considered for weapon system 
DEM/VAL source selections when either of the following 
conditions exist. 

• Significant software developed during DEM/VAL is 
planned to be reused in the EMD phase. 

• Major EMD software contractors are expected to be 
selected from the DEM/VAL phase contractors. 

SOURCE: AFMC Pamphlet 800-61 

Figure 9 Guidelines to Applying the SDCE 
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Activity 

Effort 

(Person Days) 

SDCE team preparation through final RFP release 

20-80 

SDCE team proposal analysis prior to site visit (per 
offeror) 

18-24 

SDCE team site visit (per offeror) 

24-30 

SDCE team evaluation (per offeror) after site visit 

18 - 24 


Source: AFMC Pamphlet 800-61 
Figure 10 Estimated SDCE Costs 


the supporting contracting, engineering, and quality assurance directorates. The goal is 
to gain their support, especially in committing the required resources. [Ref. 3:pp. 46-47] 


B. ACTIVITY B: SELECT AND PREPARE TEAM 

The selection of competent and experienced individuals as members of the SDCE 
team is crucial to the success of the evaluation. While a smaller team is desired for 
efficiency, a larger team offers technical depth, coverage, and experience. Such 
potentially conflicting requirements must be adequately balanced. [Ref. 3:p. 19] 

The first step in organizing a team is to select the team leader. The team leader 
should have a minimum of 15 years of weapon systems acquisition and possess 
considerable knowledge in systems and software engineering. This person must also have 
the skills to lead small groups, and to convincingly communicate the results of the 
evaluation. 
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It is recommended that the team leader come from the PMO. This ensures the PM 


has visibility and input throughout the evaluation. It also provides a means to insure that 
the knowledge, experience, and shortfalls of the contractor’s software development 
process discovered during the evaluation are assimilated into the PMO and considered 
during the project development. [Ref. 3:pp. 48-49] 

The evaluation team is composed of two parts, a core team and a support team. 
The core team consists of the team leader and two or three additional senior members 
who are experienced with software engineering, project management, and logistics. Its 
role is to provide technical expertise to evaluate the offeror’s software development 
capability. After the preliminary source selection planning has been completed and 
specific source selection parameters and conditions have been determined, the team leader 
and PMO select individuals for the SDCE team. Selections are based on the basis of the 
qualification criteria stated below and any unique requirements for the program. [Ref. 
3:pp. 48-51] 

The combined experience of the team should cover the following areas: systems 
engineering, software engineering, program management, and logistics engineering. Team 
members should also be knowledgeable in all the CCAs and CCs that will be used in the 
evaluation. [Ref. 3:pp. 49] 

Core team members are normally members of the SSEB. They are expected to 
participate fully in the evaluation of the proposal. Recommended core team qualifications 
are listed in Figure 11. [Ref. 3:p. 49] 
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Core Team Qualification Criteria 

The Senior Systems Engineer should have experience in the systems engineering 
process. This includes experience in transformation of validated customer needs 
and requirements into a set of system products and designs (MIL-STD-499B), 
requirements definition and specification, SEMP and SEMS, and systems 
engineering requirements in the RFP. 

The Senior Software Engineer should have an extensive background and 
understanding of the software development process. This includes requirements 
analysis, design, code and test, integration, the software engineering process and 
procedures, project planning and estimation, and software engineering 
requirements used in the RFP. 

The Senior Project Manager should have experience and understanding of the 
fundamental concepts and principles of project planning, process models, project 
scheduling, and milestones. The project manager should also have experience in 
project organization and management issues, development team organization, 
project costing, and management requirements for the RFP. 

The Senior Logistics Engineer should have knowledge of all activities related to 
the development and support of software systems. The logistics engineer should 
be familiar with the software product life cycle, software support environments, 
software documentation and training, and the logistics requirements for the RFP. 

Source: AFMC Pamphlet 800-61 


Figure 11 Recommended Core Team Qualification 

The support team supplements the core team with specific skills, knowledge, and 
experience required for the evaluation. They are only called upon when their 
experience/expertise is needed. They may not stay for the duration of the evaluation. 
Support team members provide expertise in such disciplines as: software engineering, 
software management, subsystems engineering, contracting, quality assurance, 
configuration management, test, and financial management. [Ref. 3:p. 49] 
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Once the team is formed, the team leader coordinates the preparations for the 
evaluations. First, the team meets with the major divisions of the PMO to familiarize 
themselves with the subject system, the issues related to its acquisition, and any special 
considerations. A review is made of key acquisition documents such as the acquisition 
strategy, mission need statement, operational requirements document, and request for 
proposal. After the team has become knowledgeable on the project and the acquisition 
strategy, it receives training on the SDCE method. [Ref. 3:p. 52] 

C. ACTIVITY C: PREPARE PLAN AND SCHEDULE 

In the early stages of planning, it is likely that only the SDCE team leader will have 
been identified. The team leader should work closely with the PMO, the Source Selection 
Authority (SSA), and the SSEB to ensure that adequate time is reserved to conduct SDCE 
activities required during source selection. These include the allowance of sufficient time 
for the team to evaluate each proposal and to conduct site visits. Site visits are made 
between the time the competitive range is determined and best and final offers (BAFOs) 
are required. A proposed schedule is provided in the AFMC pamphlet. [Ref. 3:pp. 53-56] 

The team leader also works with the SSEB to determine how the SDCE results will 
be integrated in the source selection process. The normal approach is to use SDCE 
results as a factor under the "Technical" area. The actual approach can vary based on the 
determination of the SSA. [Ref. 3:pp. 54-58] A key decision made by the SSA is 
whether to award the contract with or without discussions. In either case, the team leader 
plans for site visits. The impact of a "no discussion" decision on the SDCE is that less 
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time may be available to conduct site visits. Site visits may not be required if sufficient 
information is gathered from the offerors to make a decision, even if discussions are 

planned. [Ref. 3:p. 60] 

Once team members are selected, detailed planning occurs. As more specifics on 
the source selection schedule become available, the core team develops a definitive SDCE 
activity schedule. It also develops evaluation standards for all CCs to be evaluated using 
the SDCE model. The SDCE activity schedule and CC evaluation standards are included 
in the source selection plan. [Ref. 3:p. 61] 

D. ACTIVITY D: TAILOR SOFTWARE DEVELOPMENT CAPABILITY 

EVALUATION, SELECT CRITERIA AND QUESTIONS 

The purpose of this activity is to define the scope of the SDCE. When tailoring the 
SDCE model to a specific acquisition, the SDCE team creates a subset of the SDCE 
model. This subset contains the CCs that are identified by the SDCE team as high risk 
areas to the project development. 

Tailoring is done in several steps. First, the SDCE team develops a software profile 
for the acquisition. To do this, the software project is defined in terms of: estimated 
software size, estimated development time, and estimated development team size. The 
evaluation team also identifies special technical areas important to the software project. 
These include the software language, tools, methods, systems/software engineering 
environment, complex integrated circuit development, open systems architecture. 
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commercial off-the-shelf software, reuse requirements, complex interface requirements, 
security/safety requirements, and portability. [Ref. 3:pp. 67-68] 

Once the proposed software has been defined, the SDCE team members use their 
knowledge and experience to identify those CCs that should be evaluated during the 
source selection. These high-value discriminators must be included in the RJFP. The 
documentation required from the offeror to evaluate these CCs are requested in the RFP. 
A site visit is only used to clarify and substantiate the information received from the 
offeror. The team cannot evaluate CCs not mentioned in the RFP. [Ref. 3:pp. 69-72] 
As mentioned in the previous chapter, each CC in the SDCE model has defined 
questions and criteria. When the SDCE team selects CCs for the SDCE evaluation, it also 
automatically selects the evaluation criteria and questions. In some cases a team may 
develop unprecedented CCs unique to the acquisition that are not contained in the SDCE 
model. For example, a SDCE team may want to evaluate a contractor’s ability to 
incorporate an unprecedented specific software technology on the proposed program. This 
is a significant undertaking. The SDCE team would work with the local SDCE Office 
to create the criteria and questions for such new CCs. [Ref. 3:p. 74] 

E. ACTIVITY E: PREPARE REQUEST FOR PROPOSAL AND 

INSTRUCTIONS 

In this activity, the SDCE team takes steps to make the offerors aware of the 
Government’s intent to use the SDCE during source selection. This is done through 
several means. The CBD announcement for the acquisition should include a paragraph 
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declaring that a SDCE would be used. This would also be stated in the RFP. The SDCE 
team would also provide information to potential offerors on the SDCE method and 
procedures during the bidder’s briefing. [Ref. 3:p. 77] 

The team develops the SDCE input to the RFP. An important part of this input is 
the Government’s request for information from the offerors. This describe how each high 
value CC is implemented and what would suffice as evidence of its use. Documents that 
describe how a CC is implemented include organization charts, job descriptions, 
implementation manuals, and internal standards. Evidence of use is provided in the form 
of minutes of meetings, evaluation reports, and metrics. This information should come 
from three to four completed or ongoing projects that are similar to the software being 
developed. [Ref. 3:pp. 75-76] 

In the RFP, each offeror is directed to provide in their offers answers to the SDCE 
model questions related to the high value CCs that will be evaluated. These questions are 
also included in the RFP. Answers can be references to other documents like the SEMP, 
SDP, or company policy and procedures. [Ref. 3:p. 75] 

The team also prepares the specific SDCE input to the General Notice to Offerors 
(GNTO) section of the RFP. It explains how the site visit will be conducted, the tentative 
schedule for site visits, the facilities and resources required by the SDCE team, and the 
types of contractor personnel who may be interviewed. [Ref. 3:pp. 78-80] 

Examples of the recommended wording for the SDCE input to the CBD 
announcement and RFP are provided in the AFMC pamphlet [Ref. 3:pp. 1-14, 1-18 - 1- 
21 ]. 
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F. ACTIVITY F: REVIEW PROPOSALS 


The SDCE team examines an offeror’s information submitted with its proposal for 
completeness and consistency. Answers to the SDCE model questions in the RFP should 
fully describe how the CCs being evaluated are implemented. All processes and 
procedures should be documented, and evidence of their use should have been submitted. 
The SDCE team also checks whether the proposed software development approach is 
consistent in the proposal, SEMP, SEMS, and SDP. After reviewing the information, the 
SDCE team performs an initial assessment of strengths, weaknesses, and risks, using the 
method described in Activity I. [Ref. 3:pp. 82-84] 

Once the initial assessment is made, clarification requests (CRs) and deficiency 
reports (DRs) are created. CRs are used to request additional information to clarify a 
process or procedure, or to show evidence of use. CRs are also used to document 
inconsistencies between the SEMS, SEMP, and SDP. When a capability or procedure is 
clearly deficient with respect to RFP requirements, it is documented in a DR. CRs and 
DRs are released to the offeror if discussions are permitted. If discussions are not 
allowed, CRs and DRs are used in the evaluation to document weaknesses in an offeror’s 
proposal. [Ref. 3:pp. 86-87] 

G. ACTIVITY G: PLAN FOR AND CONDUCT SITE VISIT 

The SDCE team plans for a three to four day site visit. The site visit has three 
purposes. First, it provides a forum for the SDCE team and the offeror to discuss and 
develop a mutual understanding of the proposed software development process. It also 
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provides a means to verify a vendor’s ability to implement its proposal. Finally, it gives 
the SDCE team the opportunity to encourage the offeror to incorporate the proposed 
processes into the SDP, SEMP, and SEMS. [Ref. 3:p.88] 

To prepare for the site visit, the SDCE team reviews the offeror’s proposal and any 
responses to CRs and DRs to identify areas that still need clarification. It then develops 
an agenda and topics of discussion for the site visit. These are sent to the contractor no 
later than two weeks before the site visit. The team also coordinates with the offeror to 
ensure that adequate facilities will be available to support the team. [Ref. 3:pp. 89-90] 

Once on site, the team briefs the offeror on the SDCE method and its overall use 
in the source selection. The team then begins discussions on the agenda items. 
Discussions are confined exclusively to the offeror’s proposal, performance history, and 
CRs and DRs. The evaluation team looks for completeness and adequacy in the offeror’s 
response. It also looks for a strong or weak level of compliance with the evaluation 
criteria. All information from the offeror is carefully recorded by the SDCE team. [Ref. 
3:pp. 92-93] 

At the end of the discussion, the SDCE team reviews any additional documentation 
provided by the offeror. It then holds a meeting to review the information collected and 
to develop a team understanding of the offeror’s processes and capability. Afterwards, 
the team prepares a feedback presentation to the offeror. This is the last event of the site 
visit. The purpose of this presentation is to communicate the team’s understanding of the 
offeror’s processes and capability. The offeror is given the opportunity to respond to this 
briefing. [Ref. 3:pp. 93] 
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H. ACTIVITY H: ANALYZE CLARIFICATION REQUESTS AND 


DEFICIENCY REPORTS 

CRs and DRs are generated throughout the evaluation process. These may be 
released to the offeror with the permission of the SSA. The offeror’s response to CRs 
and DRs must then be analyzed; when required, follow-up CRs or DRs are generated. 
[Ref. 3:pp. 96-97] 

I. ACTIVITY I: EVALUATE, SCORE, AND INTEGRATE RESULTS INTO 
SOURCE SELECTION 

In this activity, the SDCE team evaluates the information received by the offeror 
and scores the results. An offeror’s capability is scored in two areas. The first is the 
degree to which the offeror’s capabilities met the evaluation criteria. The second is the 
overall risk of the offeror’s proposed development plan. [Ref. 3:pp. 100, 105] 

Evaluations are conducted from bottom to top. The SDCE team begins at the CC 
level then consolidates the score to the FA level. Capabilities are scored as either strong, 
acceptable, or weak. Risks are scored as either high, moderate, or low. Both the 
capability and risk scores are accompanied by supporting narratives. The guidelines for 
doing this are depicted in Figure 12. [Ref. 3:pp. 114-116] 

Once an offeror’s capability and risk have been scored at the FA level, the SDCE 
team assigns a color code and risk rating to the offeror’s proposal. The color code and 
risk rating are used by the SSA or Source Selection Advisory Council (SSAC) to compare 
offerors. The SDCE uses its knowledge and experience to develop these scores. The 
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CCA Capability Roll-up Guidelines 


RATING 

DESCRIPTION 

Strong 

The majority of the CCs are strong ; 

and there are no significant weak CCs 

Acceptable 

The majority of the CCs are 
acceptable and there are no 
significant weak CCs 

Weak 

There are significant weak CCs 


FA Capability Roll-up Guidelines 


RATING 

DESCRIPTION 

Strong 

The majority of the CCAs are strong 
and there are no significant weak CCAs 

Acceptable 

The majority of the CCAs are 
acceptable and there are no 
significant weak CCAs 

Weak 

There are significant weak CCAs 


CCA Risk Roll-up Guidelines 


RATING 

DESCRIPTION 

Low 

The majority of the CCs are low risk 
and there are no significant high risk 
CCs 

Moderate 

The majority of the CCs are moderate 
risk and there are no significant high 
risk CCs j 

High 

There are significant high risk CCs 


Source: AFMC Pamphlet 800-61 


Figure 12 SDCE Roll-up Guidelines to the FA Level 

source selection scoring guidelines are listed in Figure 13. [Ref. 3:pp. 119-120] 
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Source Selection Color Codes 


COLOR 

RATING 

DEFINITION 

Blue 

Exceptional 

Exceeds specified performance of 
capability in a beneficial way to 
the Government, and has no 
significant weaknesses. 

Green 

Acceptable 

Meets evaluation standards, and any 
weaknesses are readily correctable. 

Yellow 

Marginal 

Fails to meet evaluation standards, 
however any significant 
deficiencies are correctable. 

Red 

Unacceptable 

Fails to meet minimum requirements 
of the RFP, and the deficiency is 
uncorrectable without a major 
revision of the proposal. 

Proposal Risk Ratings 

SYMBOL 

RATING 

DEFINITION 

L 

Low 

Has little potential to cause 
disruption of schedule, increase in 
cost, or degradation of 
performance. Normal contractor 
effort and normal Government 
monitoring will probably be able to 
overcome difficulties. 

M 

Moderate 

Can potentially cause some 
disruption of schedule, increase in 
cost, or degradation of 
performance. However, special 
contractor emphasis and close 
Government monitoring will probably 
be able to overcome difficulties. 

H 

High 

Likely to cause significant serious 
disruption of schedule, increase in 
cost, or degradation of performance 
even with special contractor 
emphasis and close Government 
monitoring. 


Source: AFMC Pamphlet 800-61 


Figure 13 SDCE Source Selection Scoring Guidelines 
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J. ACTIVITY J: INCORPORATE INTO CONTRACT 

The purpose of this activity is to gain contractual commitment for the development 
processes being proposed by the offeror. Responses to the CRs and DRs are not binding 
by themselves. They must be incorporated into key development documents, which 
include the SDP, SEMP, and SEMS to be enforceable. [Ref. 3:pp. 124-125] 

K. ACTIVITY K: CONCLUDE SOFTWARE DEVELOPMENT CAPABILITY 
EVALUATION TEAM ACTIVITIES 

At this point, the major activities of the SDCE team have been accomplished. The 
team members derive the SDCE metrics that will be used to improve the SDCE method. 
SDCE metrics include problems encountered during the evaluations, new CCs developed, 
strengths and weaknesses of the SDCE method, and recommended improvements. The 
team also properly disposes of SDCE data no longer needed. The final action of this 
activity is the disbanding the team. [Ref. 3:pp. 126-127] 

L. ACTIVITY L: CONDUCT FORMAL FEEDBACK 

Former SDCE team core members may participate in briefings to the winner and 
unsuccessful offerors. They answer questions concerning the SDCE evaluations within 
the guidelines established by the contracting officer. They also solicit feedback on the 
vendor’s experience with the SDCE. This feedback will be used to improve the SDCE 
method. [Ref. 3:pp. 128-130] 
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M. ACTIVITY M: SUPPORT PROGRAM FOLLOW-THROUGH 

This activity involves using the SDCE results for purposes other than source 
selection. This could include input to the risk management plan and to an offeror’s 
improvement plan. It can also include using the SDCE results to monitor the progress 
of the contractor, especially in those areas that were identified as weak in the contractor’s 
software development capability. [Ref. 3:pp. 131-132] 

N. SUMMARY 

The last four chapters provided the reader with a basic knowledge of SCE, SDCE, 
and their respective models. The next chapter will present the findings of two DOD 
studies on software capability evaluations. 
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VI. LITERATURE RESEARCH 


The previous chapters provided the reader with an overview of the SCE and SDCE 
methods, along with their corresponding models. This chapter presents the findings of 
two DOD studies that compared the SCE with the SDCE’s predecessor, the SDCCR. 

A. AEROSPACE CORPORATION STUDY 

In June 1992, the Aerospace Corporation researched and published the report Pre- 
Award Survey Method for Software Acquisition at Space Systems Division in response 
to a USAF Space Systems Division (SSD) tasking [Ref. 16]. Although unable 
to obtain an actual copy of the study, the researcher was able to obtain information on the 
study by interviewing one of its authors. 

In the interview, one of the authors of the Aerospace Corporation report explained 
that Aerospace Corporation is under contract with the SSD to conduct source selection 
software capability evaluations. The SSD directed the Aerospace Corporation to conduct 
a study to determine which evaluation, the SCE or SDCCR, should be adopted to assess 
the software development capabilities of offerors during the source selection of SSD 
programs. The study team examined five to six software intensive satellite and radar 
programs and identified areas of risks to the software development project. The study 
team then determined which of the two methods, the SDCCR or SCE, was better at 
evaluating these areas. 
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The study concluded that, although the SDCCR had weaknesses, the initial 
recommendation was to use the SDCCR. The long term recommendation was to 
combine the best of both approaches and create a new evaluation method. [Ref. 16:p. 46]. 

In the same interview, the author cited the following reasons for the study’s 
conclusions: 

• "The SCE assumes that software is developed in a vacuum." The SCE had a 
narrow focus of software development. It did not address key issues, such as systems 
engineering, and program specific technologies and tools that have an impact on software 
development. The SDCCR covered these areas. 

• While answers to the "yes/no" type questionnaire used by the SCE indicate 
whether a process exists, they do not provide enough information to determine the 
adequacy of the process. The SDCCR essay type questions did. 

• In the SCE, a process was only judged to be adequate if it was already in 
existence and its procedures and proof of past use were well-documented. No 
consideration was given to new processes being proposed for the project. For the 
SDCCR, the SDCCR team uses its knowledge and experience to evaluate new processes 
being proposed by the contractor for the specific project Some consideration may be 
given if warranted. 

• The SDCCR did have weaknesses. The structure of the SDCCR model made it 
difficult to tailor the method to a specific acquisition. The SDCCR also did not address 
important areas covered in the CMM, such as defect prevention, metrics, and process 
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improvement. This is why the study recommended combining the strengths of the 
SDCCR and SCE to create a new evaluation method. 

B. INSTITUTE FOR DEFENSE ANALYSIS (IDA) STUDY 

In 1992, the Defense Advanced Research Projects Agency (DARPA) (now known 
as the Advanced Research Projects Agency (ARPA)) tasked the Institute for Defense 
Analysis (IDA) to analyze policy issues involving a wider DOD application of the SEI 
software process maturity model [Ref. 16:p. v]. The IDA study focused on 
implementation issues associated with mandating software process assessments and 
software capability evaluations DOD-wide, the availability and adequacy of data to 
determine the benefits and effectiveness of process maturity, and a comparison of the 
maturity model and methods with similar techniques. IDA drew on its experience in 
performing SCEs in support of what was then called the Strategic Defense Initiative 
(SDI). It also used the results of a survey of 123 organizations within the SPA/SCE 
community. [Ref. 16:pp. v, 80] 

A summary of the highlights of the IDA study includes the following: 

• Although the SCE had weaknesses, it helps to promote desired process 
improvement, and SCE interviews verify an organization’s current software development 
process capability. [Ref. 16:pp. 11, 51]. 

• Definitive data do not exist to quantify the benefits and effectiveness of process 
maturity. Anecdotal evidence of organizations experiencing significant positive return on 
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investment (ROI) by increasing their process maturity levels supports the pursuit of 
process improvement based on the CMM [Ref. 16:p. 32], 

• Sixty-eight percent of the contractors who had responded to the survey and had 
been subjected to more than one SCE, felt that SCEs were useful in evaluating a 
contractor’s software process capability [Ref. 16:p. 80]. 

Some of the weaknesses of the SCE and SPA identified by the study include the 
following: 

• During source selection, only existing, in-place processes were evaluated by the 
SCE team. Software engineering process improvements proposed by the contractor for 
the software development project had little bearing on the SCE results. There may be 
important differences between the software engineering processes being proposed for a 
specific acquisition and the processes being evaluated by the SCE team. [Ref. 16:p.20] 

• Several areas important to software development are not covered in the CMM, and 
therefore, are not evaluated in the SCE. These areas include human resources, software 
engineering methods and tools, and product specific technologies. [Ref. 16:p. 20] 

• Results between SCEs and SPAs can differ as much as one or two maturity levels. 
The difference can be attributed to differences in interpreting SEI rating criteria, 
differences in type of projects, and differences in SPA and SCE team experience. [Ref. 
16:p. 36] 

The IDA study team reviewed the SDCCR and compared it with the SCE. The 
IDA reported the following as strengths of the SDCCR: 
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• The SDCCR method focused on the software development processes and 
capabilities proposed for a specific acquisition program. 

• The SDCCR used a set of essay questions as one means of obtaining information 
from the contractor. This allows an evaluation to occur when no "discussions" are 
allowed; that is, the winner of the contract will be based solely on the evaluation of the 
proposals submitted by the offerors. Contractors also report that although answering the 
questions is costly, the exercise provides valuable insight to the issues important to the 
Government and into their own development practices. 

• The SDCCR site visits allowed contractors to explain and rationalize their 
proposed software development approach. 

• The SDCCR evaluated areas not addressed by the SCE such as systems 
engineering. [Ref. 16:p. 95] 

The report identified the following weaknesses of the SDCCR: 

• The SDCCR evaluation team relied on an offeror’s documentation to verify that 
the proposed practices have been used on previous projects. No individual interviews 
were conducted to verify that the documents accurately reflect the processes being used. 

• The SDCCR contained 450 essay questions. It could cost a contractor as much 
as $500,000 to answer the SDCCR questions. 

• The criteria for scoring or rating answers to the SDCCR questions were not 
completely defined. 

• The SDCCR results provided limited input towards process improvement. 
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• There were no guidelines to tailor the SDCCR to a specific acquisition. [Ref. 
16:p.95] 

It is important to note that these studies were based on older versions of the SCE 
and CMM [Ref. 16:p. 5]. They also reflect the strengths and weaknesses of the SCE and 
SDCCR, both of which formed the baseline for developing the SDCE. 

In the interviews conducted by the researcher, the discussions on the strengths and 
weaknesses of the SCE and SDCE centered around these same issues mentioned by the 
studies. In an interview with the researcher, an SEI representative raised the additional 
issue concerning the uncertainty surrounding the newly developed SDCE method versus 
the proven benefits of the SCE. 

C. SUMMARY 

This chapter presented the findings of the Aerospace Corporation and IDA studies 
on software capability evaluations. Strengths and weaknesses can be categorized in the 
following areas: scope of the evaluations, credit for new processes, questionnaires, 
tailoring to a specific acquisition, organization versus project focus, conducting site visits, 
process improvements, possible sampling errors, possible differences between SPA and 
SCE, and history of success. These areas are explained in the following chapter. The 
next chapter also compares the SCE and SDCE in these areas. 
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VII. COMPARISON AND ANALYSIS 


In this chapter a comparison of the SCE and SDCE in the areas identified in the 
previous chapter is made. The issues involving each area, data pertinent to the issues, and 
conclusions will be discussed. 

A. COST OF THE EVALUATIONS 

1. Issue 

PMs have finite resources. An issue in deciding which evaluation to use is the cost 
to conduct an SCE or SDCE. SEI and ASC/AFMC measure the cost of their respective 
evaluations in terms of human resources, or person days. 

2. Research Findings 

Figure 14 depicts SEI’s estimate to conduct an SCE for a single source selection 
with three offerors within the competitive range. The estimated cost is approximately 115 
person days. 

ASC/AFMC’s estimate to conduct an SDCE for single source selection with three 
offerors within the competitive range is also shown in Figure 14. The estimated cost 
ranges from 200 to 314 person days. 

These figures show that the cost of conducting an SDCE is almost double or triple 
the cost of conducting an SCE. This disparity in costs can be attributed to the following: 
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SCE Phase 

Effort 

(Person Days) 

SCE Information Gathering 

5.25 

REP Preparation 

3.0 

SCE Training 

25.0 

SCE Findings Preparation 

5.0 

Site Visits 

75.0 

Contractor Debriefs 

1.5 

| SCE Total for Three Site Visits 

114.75 

SDCE Activity 

Effort 

(Person Days) 

SDCE team preparation through final RFP release 

20 - 80 

SDCE team proposal analysis prior to site visit 

54 - 72 

SDCE team site visit 

72 - 90 

SDCE team evaluation after site visit 

54 - 72 

SDCE Total for Three Site Visits 

200 - 314 


SCE Source: CMU/SEI-93-TR-18 
SDCE Source: AFMC Pamphlet 800-61 


Figure 14 SCE Versus SDCE Manpower Costs 

• The SCE team is composed of four to six personnel [Ref. 5:p. 22], An SDCE 
team is composed of a core and support team. The core team has four personnel; the 
support team could have up to eight. [Ref. 3:pp. 50-51] An SDCE team could therefore 
be double or triple the size of an SCE team. 
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• Figure 14 shows that SDCE teams expend a lot of effort on proposal analysis. 
The SDCE uses essay type questions, which provide more contractor information to 
evaluate than the SCE "yes/no" questions. 

• Figure 14 also shows that an SDCE team applies a considerable amount of labor 
for evaluation activities after the site visits have been conducted. SCE activities end once 
the SCE team delivers its findings to the SSEB. For the SDCE, core team members 
participate in the evaluation of all offerors as members of the SSEB. 

The figures provided above are estimates. Many additional factors can affect the 
actual costs. These include the complexity of the system being developed, the complexity 
of the source selection, the team’s prior experience in conducting software capability 
evaluations, the number of people on the team, and the personnel used on the team 
(contractor, Government civilian, or military) [Ref. 3:p. 62] [Ref. 10:pp. 3-54 - 3-56]. 

3. Conclusions 

The findings show that due to the greater volume of information to analyze prior 
to the site visits, the potential for an SDCE team to be double or triple the size of an SCE 
team, and participation of SDCE team members in the SSEB, the cost of conducting an 
SDCE is significantly greater than that of an SCE. 

B. SCOPE OF THE SDCE AND SCE 

1. Issue 

Mission Critical Computer Resources (MCCR) refers to the totality of computer 
hardware and software that is integral to a weapon system, along with the associated 


87 


personnel, documentation, supplies, and services [Ref. l:p. 1]. As the definition of 
MCCR implies, there are many critical areas that affect software development. These 
include such issues as systems engineering, human resources, tools, and technology. An 
issue a PM must consider is whether the scope of each evaluation method is broad 
enough to cover these areas. 

2. Findings of Previous Studies 

According to the IDA study, the SDCCR covered areas not addressed by the SCE 
to include systems engineering and tools [Ref. 16:p. 95]. In the interview mentioned 
earlier, an author of the Aerospace Corporation study came to the same conclusion. 

3. Research Findings 

The SDCE inherits the broad scope of the SDCCR. In fact, the SDCE model is 
composed of critical capabilities that have historically been high-risk areas during 
software development [Ref. 3:p. 12]. For example, it addresses the systems engineering 
activities that impact the software development process. 

Systems Engineering is important in the development of a weapon system. The 

Defense Systems Management College defines systems engineering as follows: 

Systems Engineering is a management function which controls the total system 
development effort for the purpose of achieving an optimum balance of all system 
elements. It is a process which transforms an operational need into a description of 
system parameters and integrates those parameters to optimize overall system 
effectiveness. [Ref. 17] 

It is through systems engineering that software requirements are developed. 
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Problems occur when software requirements do not match the system’s 
requirements. The case of the Cheyenne Mountain Upgrade (CMU) development can 
serve as an example of this point. The CMU program is the modernization of command 
and control subsystems at Cheyenne Mountain, which serves as the command center for 
the North American Aerospace Command (NORAD). These subsystems provide critical 
strategic surveillance and attack warning to US and Canadian leaders. [Ref. 18] 

There were two systems engineering related software problems with CMU. First, 
because the USAF did not adequately define the system’s requirements, software was 
developed for a specific hardware platform. This presents a significant software 
portability problem should the USAF decide to upgrade the computer hardware. Also, 
because of the lack of adequate system requirements, the subsystems are not expected to 
properly interface with each other. [Ref. 18] 

Unlike the SCE, the SDCE places heavy importance on systems engineering. Those 
areas of systems engineering that affect software development are included in the SDCE 
model in the Systems Engineering FA. A senior systems engineer is part of the core 
evaluation team. Finally, emphasis is placed on comparing software development 
processes with key systems engineering documents such as the SEMP, SEDS, and SEMS. 

Human resources is another area addressed by the SDCE and not by the SCE. 
Human resources are important because software development is largely labor intensive. 
The outcome of a project depends on the abilities of the development team’s personnel. 
The Human Resources CCA addresses an offeror’s capability to recruit, retain, and 
efficiently allocate skilled labor to support the software development. [Ref. 3:p. 35] 
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Another area of concern in the development of embedded software is specific 
technologies required by the project. This is illustrated by the development of the BSY-2 
software. The BSY-2 is a combat weapon system designed to detect, track, and launch 
weapons for the US Navy’s new attack submarine, the Seawolf. One of the software 
problems identified by GAO on the BSY-2 was the incomplete data base management, 
design which may degrade system performance [Ref. 19]. The CCA Data Base 
Management under the Program Specific Technologies FA of the SDCE model evaluates 
an offeror’s ability to develop a complex data base system. [Ref. 3:p. 39] 

A project’s unique technology may not be in the Program Specific Technologies FA 
of the SDCE model. The SDCE method, however, allows the PMO to develop questions 
and evaluation criteria for a specific technology and add it to the evaluation. 

The SCE evaluates a contractor’s process capability against the CMM. The CMM 
was designed to focus on a limited set of key process areas that have been proven to 
enhance software development and maintenance capability. As a result, the CMM, and 
therefore the SCE, does not address important areas such as systems engineering, human 
resources, and specific technologies. 

The SEI acknowledges that there are important issues outside software development 
that are not addressed by the SCE. The SEI states that these areas should be evaluated, 
but done outside the context of the SCE [Ref. 10:p. 1-19]. 

In order to evaluate these important areas, many SCE users develop pseudo KPAs. 
For example, in the course of conducting an SCE to evaluate offerors for a $95 million 
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software avionics contract, the SCE team noted that the SCE did not address systems 
engineering issues [Ref. 20]. 

We must investigate key issues in hardware and systems engineering in our contractor 
selection because few of our projects involve only software development. Most of our 
application software integrates with hardware and is a modification or enhancement of 
existing software. 

The team created a hardware and systems engineering questionnaire to add these areas 
to the evaluation. 

Another example was provided in an interview between the researcher and an SCE 
user who had conducted four SCEs for the SDI program. She stated that the SDI 
program has special security requirements that an offeror must meet. These security 
requirements are not covered in the CMM, and therefore not evaluated by the SCE. She 
developed "pseudo" KPAs using the National Security Agency’s (NSA) Trusted 
Methodology method. 

The limited scope of the SCE is tied to the CMM. As stated in Chapter n, the SEI 
may expand the CMM in future versions. The researcher also discovered in an interview 
with an SEI representative that the SEI is working with industry and professional 
organizations to develop a Systems Engineering CMM and a People CMM. Ownership 
and maintenance responsibilities for these CMMs have yet to be established. 

4. Conclusion 

The SDCE team evaluates an offeror’s software development capability against the 
SDCE model. The SDCE model is composed of critical capabilities that have historically 
been high-risk areas in past software developments. The SDCE model address areas not 
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found in the CMM such as systems engineering, human resources, and specific 
technologies. As a result, the SDCE has a greater scope of evaluation than the SCE. 

C. CREDIT FOR NEW PROCESSES 

1. Issue 

Both methods review software development documents to verify that processes and 
procedures are actually implemented in accordance with an organization’s documentation. 
An issue that a PM must consider is how does the SDCE and SCE deal with new 
processes proposed by an offeror that has little or no documented history. 

2. Findings of Previous Studies 

The IDA study criticized the SCE for only evaluating existing processes. Process 
improvements newly initiated by a contractor had little or no impact on the SCE results 
because proof of implementation did not exist. This criticism was echoed by an author 
of the Aerospace study. 

3. Research Findings 

As stated in Chapter III, literature research indicates that this weakness has been 
corrected in the SCE version 1.5. SCE results have been changed to reflect the strengths, 
weaknesses, and process improvement activities at the KPA level. Ongoing process 
improvements related to the KPAs of the target process capability are evaluated during 
the site visit. The SCE team uses its knowledge and experience to determine and report 
the adequacy of ongoing process improvements to the SSEB. 
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The SDCE method also evaluates and reports on new processes. When a new 
process is proposed by the contractor for use on the project, because there is no proof of 
past implementation, a CR is generated. This new process then becomes a topic of 
discussion during the site visit. The SDCE team evaluates whether the new process is 
appropriate to the project and whether the contractor can sufficiently execute it as 
planned. 

4. Conclusion 

The importance of evaluating and reporting new processes is best stated by the 
ASC/AFMC representative who said, "If we did not give credit to new processes, we 
would still be coding software using ones and zeroes." Both methods evaluate and report 
new processes. It is important to note an important difference between how the SDCE 
and SCE address new processes. As mentioned previously, each method only evaluates 
new processes that are covered in their respective models. The SDCE model addresses 
areas not covered by the CMM such as systems engineering, human resources, and 
specific technologies. In a SDCE evaluation, a contractor can receive credit for new 
processes in these areas, where it could not in an SCE. 

D. QUESTIONNAIRES 

1. Issue 

Both methods use a questionnaire to provide information concerning an offeror’s 
software development capability. An issue a PM should consider is what are the strengths 
and weaknesses of the SDCE and SCE questionnaire. 
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2. Findings of Previous Studies 

According to the IDA study, a strength of the SDCCR was its essay type questions. 
The SDCCR questions solicited more information on an offeror’s software development 
capability than the SCE questionnaire. This allowed the SDCCR to be used in source 
selection evaluations under the "without discussions" condition. In "without discussions" 
source selections, site visits may not be conducted. However, a limited evaluation of an 
offeror’s software development capability could be conducted based on an offeror’s 
responses to the SDCCR questions. 

The IDA study identified additional benefits of the SDCCR questionnaire to 
contractors as well. The SDCCR questions used in the source selection help key 
contractors on the Government’s concerns regarding the acquisition. Contractors also 
report that answering the questions provides valuable insight into their own processes and 
capabilities that will be used on the project. 

According to the IDA study, there was a problem with the SDCCR questions. The 
criteria for scoring or rating the answers to the SDCCR questions were not completely 
defined. 

The author of the Aerospace study found the questionnaire used by the SCE to be 
a weakness. The "yes/no" response to the SCE questionnaire did not provide the detailed 
information required to determine the adequacy of the process. 

3. Research Findings 

The researcher questioned the ASC/AFMC representative about the SDCCR’s role 
in a "without discussions" source selection. He stated that many source selection plans 
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for weapon system acquisitions are written to award the contract "without discussions." 
Although these acquisitions begin with the "without discussions" condition, because 
weapon system acquisitions are complex, the SSA usually opens the source selection to 
allow "discussions" between the Government and the offerors. 

This is not always the case, however. A contract for an F-16 component was 
awarded "without discussions." The SDCCR was used during this acquisition’s source 
selection activities to evaluate an offeror’s software development capability. Since 
"discussions" were not allowed, no site visits were conducted. The offerors’ software 
development capabilities were evaluated by their answers to the SDCCR questions and 
the content of their software development plans. 

In an interview with the researcher, a contractor representative stated that his 
company has been subjected to both the SDCCR and SCE. Many in his company found 
it easier to answer the SDCCR questions than those of the SCE. The company could 
describe how a process is performed within the organization in response to a SDCCR 
question. He explained that the problem with answering the SEI maturity questionnaire 
stems from an unclear definition of a "yes" or "no" answer. For example, a question from 
the SEI maturity questionnaire asks "Does each software developer have a private 
computer-supported work station/terminal? If 99% of the workers have a terminal, does 
this call for a ’yes’ or ’no’?" The contractor representative also stated that the exercise 
of answering the SDCCR questions provided insight into the company’s software 
development capability. 
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As stated in Chapter IV, the SDCE questions are in the same essay format as the 
SDCCR. The essay type questions makes the SDCE method useful during "without 
discussions" source selections. The questions are part of the SDCE model. 

As mentioned in Chapter IV, and illustrated in Figure 5 of the same chapter, model 
criteria are defined for each SDCE question. This solves the problem identified by the 
IDA study concerning the absence of defined criteria to score the SDCCR questions. 

As mentioned in Chapter HI, the SEI acknowledges that there were problems with 
its maturity questionnaire. The questionnaire does not address some KPAs. Some 
questions are not linked to any KPA at all. The SEI has mitigated the questionnaire 
problem by switching the SCE focus from being "question-based" to "model-based." 
According to the SEI, the maturity questionnaire is used only as an input source during 
the specific preparation phase and does not have a direct impact on the SCE results [Ref. 
10:p. 2-39]. This was verified by the SCE user supporting the SDI program in an 
interview. She said that the questionnaire is hardly used. When conducting site visits, 
emphasis is placed on verifying KPAs, not the offeror’s response to the questionnaire. 

As mentioned in the previous paragraph, the purpose of the SEI’s maturity 
questionnaire during an SCE is to provide input to the specific preparation phase and has 
little impact on the SCE results. As stated in Chapter HI, the SCE uses two primary 
means of soliciting information to evaluate an offeror’s process capability. They are 
interviews and documentation reviews, not the maturity questionnaire. In addition, the 
offerors record their answers to the maturity questionnaire on an SEI assessment-recording 
form. In this form, each question has a section that solicits comments from offerors to 
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amplify their "yes/no" responses. [Ref. 22:pp. 39, 44] Based on this information, SCE 
version 1.5 eliminates the criticism that the binary format of the SEI maturity 
questionnaire does not provide enough information to determine the adequacy of the 
process. 

The SEI released a new questionnaire based on the CMM version 1.1. this year. 
In an interview, an SEI representative stated that the new questionnaire has corrected the 
problem of relating KPAs to questions experienced by the previous version of the 
maturity questionnaire. 

4. Conclusion 

The SDCE essay type questionnaire solicits more information than the SEI’s 
maturity questionnaire. Of the two methods, only the SDCE is useful in "without 
discussions" source selections. The process of answering the SDCE questions provides 
an offeror with insight into its own software development capability. The criteria for 
scoring the SDCE questions are also defined. 

There were problems of relating questions to KPAs in the previous version of the 
maturity questionnaire. The SEI has released a new maturity questionnaire that has 
corrected these problems. 

E. TAILORING 

1. Issue 

As mentioned in the previous chapters, each method uses a tailored subset of their 
respective model to evaluate an offeror’s software development capability for a specific 
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acquisition. An issue PMs must address is the ease with which the SCE or SDCE may 
be tailored to address the unique requirements of their programs. 

2. Findings of Previous Studies 

A weakness of the SDCE’s predecessor, the SDCCR, was that no guidance was 
provided for tailoring the SDCCR for a particular acquisition [Ref. 16:p. 95], In an 
interview, a member of the IDA team that conducted the study stated that some SDCCR 
teams found it too difficult to tailor the SDCCR. As a result, the SDCCR teams sent all 
450 questions for the offerors to answer. It was estimated that it could cost a contractor 
up to $500,000 to answer all the SDCCR questions [Ref. 16:p. 46], 

The author of the Aerospace Corporation echoed the same criticism in her interview 
with the researcher. She stated that the structure of the SDCCR model made it difficult 
to tailor the method to a particular acquisition. 

3. Research Findings 

In the same interview with the author of the Aerospace Corporation study, she 
stated that she was a member of the SDCE model development team that was part of the 
SDCE PAT. She has also used the SDCE model and questionnaire four times in source 
selections for aircraft radar and satellite systems. She found that it was easy to tailor the 
SDCE model for each acquisition. The manner in which the SDCE model is organized 
fixed the problem the SDCCR had of tailoring. 

The researcher questioned the ASC/AFMC representative on the use of the SDCE 
model in these acquisitions. He stated that he knew of these acquisitions. In them, the 
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Aerospace Corporation evaluation team did not complete all the activities of the SDCE 
method. Therefore, his office does not recognize these acquisitions as pilot uses of the 
SDCE method. 

In the same interview, the ASC/AFMC representative stated the SDCE model 
structure facilitates the tailoring of the SDCE method to a specific acquisition. For 
example, eliminating a FA automatically eliminates the model questions of the CCs 
associated with the FA. 

The ASC/AFMC representative also provided another explanation as to why the 
entire SDCCR questionnaire was sent to the offerors. He stated that the questions cover 
so many important risk areas that SDCCR teams wanted answers to all the questions. He 
is encountering this problem in the pilot use of the SDCE method on the TSSM program. 
According to the ASC/AFMC representative, the TSSM PMO originally identified a large 
number of CCs for use in the source selection evaluation of an offeror’s software 
development capability. These CCs encompassed approximately 800 questions from the 
SDCE model. The ASC/AFMC office is working with TSSM PMO to reduce the number 
of SDCE questions for this acquisition. 

As described in Chapter HI, the SCE has an effective method of tailoring. It uses 
the maturity levels. Once the SCE team has determined the target process capability, the 
KPAs of the corresponding maturity level and the maturity levels below it are selected 
for evaluation. 
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4. Conclusions 


The SCE is the easiest of the two methods to tailor to a particular acquisition. 
Tailoring is based on the target process capability and maturity levels. Although the 
Aerospace Corporation’s use of the SDCE model is not recognized by ASC/AFMC as a 
pilot use of the SDCE, it shows that the problem of tailoring associated with the SDCCR 
model structure has been corrected in the SDCE model. A problem facing SDCE users 
now is limiting the CCs and associated questions for use in a SDCE evaluation to those 
high value discriminators that will provide the greatest insight into an offeror’s software 
development capability. 

F. PROJECT FOCUS 

1. Issue 

The PM is responsible for the success of his program. His primary focus is on 
those factors affecting his program. An issue a PM should consider when choosing 
between the two methods is which one has a greater focus on the software development 
risks of his program. 

2. Findings of Previous Studies 

According to the IDA study, a strength of the SDCCR method was that it focused 
on the proposed processes and capabilities that were proposed by an offeror to develop 
the software project. It did this by scrutinizing the software development plan submitted 
by an offeror with its proposal. [Ref. 16:p. 46] 
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3. Research Findings 

As mentioned in Chapter V, the SDCE, like the SDCCR, focuses on the processes 
and capabilities that are required to successfully develop the software for the specific 
project. Based on a project’s unique characteristics, the SDCE team identifies those CCs 
that are crucial to the success of the software development project. The vendor is 
evaluated on its ability to perform these CCs. 

Members of the software industry and Government have criticized the SCE for 
focusing more on organizational processes and capabilities rather than those required at 
the project development team level [Ref. 3:p. 12] [Ref. 13:p. 1]. For example a concern 
of the Aerospace Industries Association (AIA) is: 

While the Government must validate that a company uses the processes it professes, it 
must also focus the examination on how policies, practices and procedures would be 
implemented in the project under evaluation, with attention to what is unique on a 
program and to a selected contractor’s entire development team. [Ref. 12:p. 2] 

As illustrated in Chapter III, the SCE version 1.5 does focus on the processes required 
for the proposed software development, but within the context of the CMM and the target 
process capability. Based on the project’s requirements, the SCE team determines a target 
process capability an organization must possess to successfully develop the project’s 
software. An offeror is only evaluated on the KPAs of target process capability and the 
maturity levels below it. 

4. Conclusions 

Both methods focus on the processes and capabilities required to develop a 
particular project’s software. As discussed earlier in this chapter however, the SDCE has 
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a greater scope than the SCE. The SDCE therefore focuses more on the processes and 
capabilities required to develop a project’s software. 

G. CONDUCTING SITE VISITS 

1. Issue 

Each method uses a different approach in conducting site visits. An issue a PM 
should consider is which of the two approaches is better. 

2. Findings of Previous Studies 

The IDA study identified that the lack of individual interviews was a weakness of 
the SDCCR. Individual interviews are required to verify that the documentation 
accurately reflects the processes being used. 

The results of the survey used in the study indicated that 68 percent of the 
contractors who have been subjected to more than one SCE believed that SCEs are a 
useful method for the Government to select and monitor contractors. 

The IDA study also identifies the SDCCR site visit as a strength of the method. 
"The SDCCR includes contractor site visits, which allow contractors to explain and 
rationalize the software development approach they have proposed for the acquisition 
program." [Ref. 16:p. 46] 

3. Research Findings 

There are differences in the way site visits are conducted between the two methods. 
This stems from the different views taken by the SDCE and SCE on the purpose of the 
site visit. 
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As discussed in Chapter III, the site visit allows the SCE evaluation team to collect 
information on the offeror’s process capability through interviews and documentation 
review. Personnel at the organizational level down to the development teams of those 
projects selected for review are interviewed individually by the entire SCE team. At first 
glance, this setting may resemble an interrogation, but SCE teams are trained to place the 
interviewee at ease. 

As one SCE user described in an interview, the SCE team interviews an individual 
on a topic or several topics. The interviewee’s responses are then compared with 
documentation and information collected from other interviews. Discrepancies may show 
that processes do not exist, or may be established but are not followed or understood. 

For the SDCE, one of the goals of the site visit is not only to address software 
development issues, but to explore these issues in a "positive, team building atmosphere" 
or non-adversarial manner [Ref. 3:p. 89]. Specifically, a contractor representative, who 
was a member of the SDCE PAT, said that the SDCE was designed to avoid the 
interrogation-like atmosphere of the SCE. 

One of the purposes of a SDCE site visit is to provide "a forum for the SDCE team 
and an offeror to discuss the proposed capability and processes, in an open dialogue, with 
the objective of reaching a mutual understanding of the offeror’s capability in terms of 
processes, resources, experience, skills, tools, and technology" [Ref. 3:p. 88]. Because 
of this, the offeror selects the team members who will represent the company during the 
site visit, not the SDCE team. Also, before leaving the offeror’s site, the SDCE team 
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presents its understanding of the proposed development processes and capabilities, then 
gives the offerer a chance to respond. 

There are no data to suggest that one site visit approach is better than the other. 
As mentioned previously, the SEI claims that the SCE site visit is a proven and effective 
method of investigation in its publication. In an interview with the researcher, the 
ASC/AFMC representative claimed the SDCCR/SDCE site visit method is also effective. 
He stated that an experienced SDCE team can quickly discern whether a process is 
adequate during the site visit He pointed to the successful use of the SDCCR on over 
30 weapon system source selections. For example, the SDCCR was successfully used on 
several source selection evaluations for contracts dealing with the USAF’s next generation 
fighter, the F-22. 

As discussed in previous chapters, two common factors to both site visit methods 
are the site visit planning and the selection of the evaluation team. Effective site planning 
identifies the areas to be evaluated, the information required, the documentation to 
request, and the questions to ask. This maximizes the amount of useful information 
received during the short duration of the site visit. Staffing the evaluation team with 
knowledgeable and experienced personnel allows the team to conduct an effective 
evaluation whether using individual interviews or a discussion format. 

4. Conclusion 

When properly staffed with knowledgeable personnel, the SCE or SDCE team can 
obtain information on an offeror’s software development capability. The strength of the 
SDCE site visit is that it allows the offeror a chance to respond to the SDCE team’s 
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findings to ensure information provided by the offeror was not misinterpreted or missing. 
The strength of the SCE site visit is that it conducts individual interviews to verify that 
the documentation accurately reflects the processes being used, and to evaluate the level 
of understanding key personnel have of the software development process. 

H. PROCESS IMPROVEMENT 

1. Issue 

The effects of the SDCE and SCE on the software industry’s process improvement 
activities is also an issue. 

2. Findings of Previous Studies 

The IDA study found the SCEs were useful in promoting desired process 
improvement. IDA supports this finding by the survey results. As mentioned previously, 
68 percent of the organization who were subjected to more than one SCE believed that 
SCE are useful in identifying the strengths and weaknesses of a contractor software 
development process. The SCE findings could be used as input to an organization’s 
process improvement plan. 

3. Research Findings 

As mentioned in Chapter ID, one of the benefits of the SCE is that it promotes 
software process improvement. As Government agencies use the SCE during source 
selection, contractors must adopt the CMM based process improvement plans in order to 
stay competitive for Government contracts. Because the SCE is based on the CMM, 
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offerors have a defined method of allocating scarce resources to process improvement 
activities, in order to do better in future evaluations. 

An example of this is the Equipment Division at Raytheon. It used the CMM and 
SPA to implement process improvement to increase its maturity level from two to three. 
Raytheon credits its high maturity level as the deciding factor in winning two Government 
contracts. With the increased use of SCEs by the Government, Raytheon has placed great 
emphasis on process improvement. [Ref. 21] 

A concern of the software industry is that as more evaluations are conducted on an 
organization, process improvement plans might become less stable as the company 
constantly reprioritizes resources in an attempt to fix the deficiencies found from the latest 
evaluation [Ref. 12:p. 2]. This is not a problem for those offerors using CMM-based 
process improvement activities when being evaluated by the SCE. Since the SCE 
evaluates how well an offeror conforms to the CMM, the results could be viewed as an 
outside assessment of the company’s process improvement activities. The SCE results 
can easily be incorporated into the organization’s process improvement plan. 

Promoting process improvement is not a stated benefit of the SDCE [Ref. 3:pp. 4- 
5]. The SDCE is not tied to any process improvement methodology. Although the SDCE 
incorporated aspects of the CMM, the maturity levels were eliminated. With the 
elimination of the maturity levels, an offeror has no way of prioritizing allocation of 
resources to prepare for the SDCE. With the SDCE model containing 117 CCs 
incorporated into 37 CCAs, which in turn are grouped into six FAs, it is difficult for a 
contractor to prioritize process improvements to do better during an SDCE. 


106 



An SEI representative told the researcher in an interview that he was concerned that 
the SDCE model might reverse the process improvement gains realized in the software 
industry. The CMM is a proven model in assisting an organization in obtaining higher 
levels of process maturity. Contractors may be tempted to abandon process improvement 
activities in favor of satisfying the requirements of future SDCE evaluations. 

During an interview with a contractor representative who served as a member of the 
SDCE PAT, it was stated that the SDCE should have adopted the CMM maturity levels. 
His company does business with the other Services, not just the USAF (the only Service 
to use the SDCE). In order to stay competitive, his company must pursue process 
improvements using the CMM. In his opinion, the SDCE should have taken the SCE as 
a base and added other areas such as systems engineering. It would have been easier for 
his company to contend with just an expanded SCE than two different evaluations. 

Although the offeror is encouraged to incorporate SDCE results into its process 
improvement plans, this task may not be easily done. As previously discussed, the SDCE 
model contains areas that are not addressed in the CMM, and deficiencies in these areas 
may not be readily incorporated into an organization’s process improvement plans. With 
the elimination of the maturity levels, an area identified as weak by the SDCE may be 
a KPA that is above the organization’s maturity level as denoted by a CMM/SPA. 

In an interview with the researcher, the ASC/AFMC representative said that the 
SDCE was designed strictly as a capability evaluation, not as a method of promoting 
process improvement. As an analogy, he indicated that the SEI’s SPA was not an 
evaluation but a process improvement tool. While the SDCE method does examine an 
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organization’s process improvement activities, it only evaluates those being used on the 
project. 

4. Conclusion 

A strength of the SCE is that it promotes process improvement within the software 
industry. The SDCE is strictly an evaluation tool and was not designed to promote 
process improvement. 

I. SAMPLING ERROR 

1. Issue 

Another AIA concern regarding the SCE is the application of a single maturity 
rating to the various divisions or groups within a corporation when multiple projects are 
evaluated. The issue being raised is the accuracy of the SCE results based on an 
evaluation of a sample size of three or four software projects. [Ref. 12:p. 2] 

2. Research Findings 

The SCE team draws conclusions on an organization’s software development 
process by examining three or four of the organization’s completed or ongoing projects. 
According to the SEI, "Using its collective professional judgment and a consensus 
decision making process, the SCE team puts together its findings from individual projects 
to create a set of overall, organization-level findings" [Ref. 10:p. 2-40]. 

One interviewee who had conducted four SCEs stated to the researcher that a 
disadvantage to the SCE is that only a small sample of an offeror’s projects are examined. 
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KPAs may be evaluated as weak for these projects, but may indeed be well-implemented 
throughout the organization overall. 

The SEI is aware of this possible shortcoming and provides the following guidance 
to SCE teams: 

Findings should be specific to the point where they identify the cause for a strength or 
weakness, but not so specific that the finding places the team in a corner by failing to 
consider exceptions that may exist within an organization. SCE teams must remember 
they are evaluating a subset of the total projects ongoing at a site as a proxy for 
predicting the organization’s capability to do a specific project, and exceptions may exist 
because of this process. [Ref. 10:p. 2-40] 

The SCE method heavily depends on the SCE team members’ experience and 
knowledge to accurately estimate the process capability of an organization based on the 
small sample of projects being evaluated. However, there exists a possibility that the 
software development capabilities demonstrated by the projects that were evaluated do not 
represent the organization as a whole. 

3. Conclusions 

A potential weakness of the SCE method is that the team draws a conclusion on the 
software development capability of an organization by evaluating three to four of the 
organization’s completed or ongoing projects. While these conclusions may reflect the 
process capability of the projects that were evaluated, it may not necessarily represent 
the entire organization. The organizational software development capability may actually 
be stronger or weaker than those of the projects evaluated by the SCE team. 
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J. DIFFERENCES IN SPA AND SCE RESULTS 


1. Issue 

Another issue concerning the validity of SCE results stems from possible differences 
between the SPA and SCE results. 

2. Findings of Previous Studies 

The IDA study found that there were differences between SCE and SPA results. 
For example, the USAF reviewed two source selections encompassing 14 contractor site 
visits. The SCE findings indicate that approximately 20 percent of the contractors were 
a Maturity Level 3 as compared to the contractors’ SPA based claims that approximately 
60 percent were Maturity Level 3 organizations [Ref. 16:p. 36]. The IDA team attributed 
these differences "to many reasons: different interpretations of the SEI rating criteria, 
different projects being evaluated, different levels of experience between teams, and so 
forth." [Ref. 16:p. 36] 

3. Research Findings 

The SEI recognizes that differences between the SCE and SPA results can occur and 
emphasizes the causes of this problem in its publications. For instance, the SEI assumes 
that contractor personnel have no motive to mislead SPA team members who, with the 
full support of management, are dedicated to process improvement to raise the 
competitiveness of the company. In a SCE, the SEI assumes that the evaluated 
organization will make every attempt to put its process capability in the best possible 
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light. SCE results could therefore be higher. The SCE team must make every attempt 
to verify the facts concerning an offeror’s process capability. [Ref. 10:p. 1-21] 

Much of the evidence to support the CMM and process maturity is based on the 
positive results experienced by organizations as they improve to the next maturity level. 
In these cases, improvements in maturity level were verified using SPA not SCE results. 
The Software Engineering Division of Hughes Aircraft serves as an example 
[Ref. 22]. Hughes Aircraft’s Software Engineering Division reported a cost 
savings of two million dollars a year by improving from Maturity Level 2 to Maturity 
Level 3. 

A problem arises when SCE results differ from SPA results. Suppose an SCE was 
performed on the Software Engineering Division of Hughes Aircraft and the results 
indicate that the organization was at Maturity Level 2. One could conclude that the SPA 
results were wrong, and therefore an organization need not improve to a higher level of 
maturity to realize significant cost savings. This could cast doubt on the CMM and 
process maturity concept. One could also conclude that the SCE results were wrong, 
thereby presenting an inaccurate picture of Hughes SED’s software development 
capability. 

To address the problem of dissimilar results for the SPA and SCE, the SEI has 
combined the SCE and SPA projects into one appraisal project based on the CMM. The 
CMM Based Assessment (CBA) project will develop a common rating framework to 
describe an approach to determine and report process maturity for all CMM based 
appraisals. It will also provide guidance and standards for performing appraisals to ensure 
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the accuracy and consistency of results. The framework will have two algorithms: one 
for determining the extent to which an organization or project satisfies each subprocess 
area and another to determine an organization’s maturity level. SEI has not published a 
schedule of when these products will be available. [Ref. 23] 

In an interview with the researcher, a SEI representative stated that the SEI no 
longer conducts training on the SCE method. All SCE training is being conducted by a 
contractor. The SEI maintains ownership of the SCE method and training to ensure SEI 
standards are maintained. 

4. Conclusion 

The SEI acknowledges that SCE results may differ from those of an SPA even 
though they are both based on the CMM. When this occurs, the validity of the SCE 
results may be questionable. The SEI has created the CBA project to correct this 
problem. 

K. RECORD OF SUCCESS 

1. Issue 

As stated in Chapter I, modem weapon systems rely heavily on their computer 
hardware and software to operate in a manner in which they are designed. An issue 
important to the PM is which software capability evaluation has a proven history of 
success. 
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2. Findings of Previous Studies 

The IDA study cited anecdotal evidence of organizations experiencing significant 
positive ROI by increasing their process maturity levels. These cases were used to 
support the pursuit of process improvement based on the CMM. They were the 
following: 

• Hughes Aircraft’s Software Engineering Division reported a decrease in 
development risks as it moved from Maturity Level 2 to Maturity Level 3. Hughes used 
the Cost Performance Index (CPI) as an indicator of risk costs. CPI is calculated by 
dividing the budgeted cost of work performed (BCWP) by the actual cost of work 
performed. In July 1987, CPI was .94, which means actual costs exceeded budgeted costs 
by 6 percent. After implementing process improvements, CPI had increased to .97. This 
indicates that actual costs exceeded budgeted costs by only three percent, a drop of 50 
percent. The estimated cost savings to Hughes was two million dollars annually. [Ref. 
22 ] 

Ratheon’s Equipment Division also used the CMM and SPA to increase its maturity 
level from two to three. It reports the elimination of rework costs at a saving of $15.8 
million. Many software projects are being delivered below budget and ahead of schedule. 
In one case, this earned Raytheon a $9.6 million schedule-incentive payment. Raytheon 
also reports a return on investment (ROI) of 7.7 to 1 (a $4.48 million return on a $0.58 
million investment) for implementing process improvement. [Ref. 21] 
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3. Research Findings 

As stated in Chapters II and IE, the SCE and CMM were developed in the late 
1980s. The current versions of the SCE and CMM represent approximately eight years 
of continuous improvements from their original versions. These improvements are based 
on lessons learned and feedback from the Government and software industry. In an 
interview, the SEI representative said that the CMM is one of the most widely used 
models in the software industry worldwide. 

As stated in Chapter IV, the SDCE was developed by combining the strengths and 
correcting the weaknesses of two proven evaluation methods, the SDCCR and SCE. The 
model used by the SDCE is a collection of critical capabilities that have historically been 
areas of high risk in software development. While these two factors may indicate that 
the SDCE is an effective means of evaluating an offeror’s software development 
capability, the SDCE method is still unproven. Pilot tests of the SDCE are scheduled for 
next fiscal year. 

4. Conclusions 

The current versions of the SCE and CMM have a proven history of success. 
Although the SDCE has the potential for being an effective method for evaluating an 
offeror’s software development capability, it is still unproven. It will continue to remain 
so, until the pilot tests of the SDCE are completed next fiscal’year 
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L. Summary 

This chapter addressed the issues concerning the use of the SDCE and SCE in the 
source selection of a software intensive weapon system. Based on the previous studies 
and the findings of this research, conclusions were developed for each area. The 
following chapter will summarize these conclusions. 
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VIE. SUMMARY AND RECOMMENDATIONS 


In the previous chapter, conclusions were developed from the research findings in 
various areas. This chapter uses these conclusions to answer the research questions. The 
secondary research questions are addressed first, followed by the primary question. 
Recommendations will be made. And finally, areas of further research are identified. 

A. ANSWERS TO THE SECONDARY RESEARCH QUESTIONS 

All the secondary research questions have been answered in previous discussions. 
They are summarized in the following paragraphs. 

1. What is the Capability Maturity Model? 

This question was answered in Chapter II. The CMM represents a limited 
collection of KPAs that have been shown to enhance software development and 
maintenance capability. By focusing on this limited set of KPAs and working 
aggressively to achieve them, an organization can steadily improve its process capability. 
The CMM is a tool to assist developers in gaining control of their software development 
process and move towards continuous process improvement. 

2. What is a Software Capability Evaluation (SCE) and how is it used 
during source selection? 

This question was answered in Chapter III. The SCE is a method developed by the 
SEI for evaluating the software development process of an organization. The SCE’s role 
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in the source selection is to support the acquisition of software by assessing an offeror’s 
software process capability. The SCE identifies the strengths, weaknesses, and process 
improvement activities of an offeror. The SCE results are incorporated into the source 
selection evaluation. 

3. What is the Software Development Capability Evaluation Model? 

This question was answered in Chapter IV. The SDCE model is a collection of 
critical capabilities that have historically been high-risk areas during software 
development. The model contains the evaluation questions and criteria for each critical 
capability. 

4. What is the Software Development Capability Evaluation and how is it 
used during source selection? 

This question was answered in Chapters IV and V. The SDCE is a method used 
during source selection to evaluate an offeror’s software engineering and management 
capabilities. It also evaluates an offeror’s systems engineering capabilities which directly 
impact software development. The primary purpose of the SDCE is to increase the 
probability of selecting an offeror capable of successfully developing software that meets 
all requirements within cost and schedule constraints. 

B. THE PRIMARY RESEARCH QUESTION 

The primary research question is "What are the strengths and weaknesses of the 
SEI’s Software Capability Evaluation and ASC/AFMC’s Software Development 
Capability Evaluation, that PMs should consider when deciding which method to use on 
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their programs to evaluate a contractor’s software development capability during source 
selection?". To answer this question, the conclusions that were analyzed and discussed 
in depth in the previous chapter will be categorized as strengths or weaknesses for each 
method. A summary of the strengths and weaknesses of each method is depicted in 
Figure 15. 

SDCE SCE 

STRENGTHS STRENGTHS 

* LARGER SCOPE * * LOWER COSTS 

* ESSAY-TYPE QUESTIONS * INDIVIDUAL INTERVIEWS 

* PROJECT FOCUS * PROCESS IMPROVEMENT 

* SITE VISIT * EASIER TO TAILOR 

* PROVEN METHOD 

WEAKNESSES WEAKNESSES 

* MORE EXPENSIVE * SMALLER SCOPE 

* DIFFICULT TO TAILOR * RESULTS BASED ON SMALL 

* PROCESS IMPROVEMENT SAMPLES 

* UNPROVEN METHOD * DOES NOT MATCH SPA 


Figure 15 A Summary of Strengths and Weaknesses 
1. SDCE Strengths 

This research shows the strengths of the SDCE method are: 

• Scope of the Evaluation - The SDCE team evaluates an offeror’s software 
development capability against the SDCE model. The SDCE model is composed of 
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critical capabilities that have historically been high-risk areas in past software 
developments. The SDCE model address areas not found in the CMM, such as systems 
engineering, human resources, and specific technologies. As a result, the SDCE has a 
greater scope of evaluation than the SCE. 

• Questionnaire - The SDCE essay-type questionnaire solicits more information than 
the SEI’s maturity questionnaire. Of the two methods, the SDCE is the only evaluation 
method that may be used in "without discussions" source selections. The process of 
answering the SDCE questions also provides an offeror insight into its own software 
development capability. The criteria for scoring the SDCE questions are also defined. 

• Project Focus - Both the SDCE and SCE focus on the processes and capabilities 
required to develop a specific project’s software. However, as discussed earlier, because 
of the SDCE’s greater scope, it focuses more on the processes and capabilities required 
to successfully develop the project’s software. 

• Site Visit - A strength of the SDCE visit is that it is conducted in a non- 
adversarial setting where the offeror is given a chance to respond to the SDCE team’s 
findings. This ensures information provided by the offeror was not misinterpreted or lost 
by the SCE team. 

2. SDCE Weaknesses 

This research shows the weaknesses of SDCE method are: 

• Evaluation Costs - The findings show that a greater volume of information is 
analyzed prior to the site visits in the SDCE than in the SCE. There is a potential for a 
SDCE team to be greater in size than that of a SCE team. Unlike the SCE, SDCE core 
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team members are also members of the SSEB. Their duties in evaluating offerors as 
members of the SSEB are part of the SDCE. As a result, the cost of conducting a SDCE 
could be double or triple the cost of conducting a SCE. 

• Tailoring - The SDCE model contains critical capabilities that have shown to be 
high-risk areas for software development. A problem facing SDCE users is limiting the 
CCs and associated questions for use in a SDCE evaluation to those high-value 
discriminators that will provide the greatest insight into an offeror’s software development 
capability. 

• Process Improvement - The SDCE is strictly an evaluation tool and was not 
designed to promote process improvement. Unlike the CMM, the SDCE model structure 
does not help an organization identify and prioritize process improvement activities. The 
stability of an organization’s process improvement plans could be continuously disrupted 
by the organization’s attempt to correct weaknesses found during the most recent SDCE 
evaluation. This could have a negative impact on promoting overall process improvement 
in the software industry. 

• Unproven Method - The SDCE was developed by combining the strengths and 
correcting the weaknesses of two proven evaluation methods, the SDCCR and SCE. The 
model used by the SDCE is a collection of critical capabilities that have historically been 
areas of high risk in software development. While these two factors may indicate that 
the SDCE is an effective means of evaluating an offeror’s software development 
capability, the SDCE method is still unproven. It will continue to remain so until the pilot 
tests of the SDCE are completed next fiscal year. 
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3. SCE Strengths 

The research shows the strengths of the SCE to be: 

• Cost of the Evaluation - For reasons cited earlier, the cost of conducting a SCE 
could be half or one-third the cost of conducting a SDCE. 

• Individual Interviews - The SCE team conducts individual interviews during the 
site visit. Individual interviews verify the information found in an offeror’s 
documentation and evaluate the level of understanding of an organization’s software 
development process by its employees. The SCE’s position is that processes not 
understood by software development personnel are usually not followed. Inconsistencies 
between the information gathered from documentation and interviews may also indicate 
a weakness in a process. 

• Process Improvement - As discussed previously, the SCE promotes process 
improvement in the software industry. Using the results of the SCE as a factor in 
selecting a contractor provides an incentive for the software industry to initiate process 
improvement in order to stay competitive for Government contracts. The CMM, on 
which the SCE is based, was designed as a tool to assist an organization in identifying 
and prioritizing software process improvement activities. A continuous movement toward 
improving process capability by the software industry will benefit almost all weapon 
system programs throughout DOD. 

• Tailoring - In comparison to the SDCE, the SCE is easier to tailor to a specific 
acquisition. The SCE method uses the maturity levels of the CMM and target process 
capability developed by the SCE team for a particular acquisition. The KPAs selected 
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for use in the evaluation are those attributed to the target process capability and the 
maturity levels below it. This ease of tailoring the SCE to a specific acquisition makes 
the method easier to use. 

• Proven Method - The SCE and CMM were developed in the late 1980s. The 
current versions of the SCE and CMM represent approximately eight years of continuous 
improvement based on lessons learned and feedback from the Government and software 
industry. Unlike the SDCE, the SCE has a proven history of success. 

4. SCE Weaknesses 

The research found the following areas to be weaknesses 

• Scope of the Evaluation - The SCE team evaluates an offeror’s software 
development capability against the CMM. The CMM is composed of a limited set of key 
process areas that have been proven to enhance software development and maintenance 
capability. Unlike the SDCE model, the CMM does not address non-software issues that 
impact the development of software such as systems engineering, human resources, and 
specific technology. As a result, the SCE has a smaller scope of evaluation than the 
SDCE. 

• Results Based on Small Samples - The SCE attempts to characterize an 
organization’s software development process capability by evaluating three to four of the 
organization’s past or ongoing projects. Each project has unique characteristics and 
requirements. The processes of the projects selected for evaluation by the SCE may not 
be indicative of the entire organization. 


122 



• SCE Results Do Not Match the SPA - The SEI acknowledges that SCE results 
may differ from those of an SPA even though they are both based on the CMM. When 
this occurs, the validity of the SCE results may be questionable. The SEI has created the 
CBA project to correct this problem. 

C. RECOMMENDATIONS 

Based on the strengths and weaknesses, the researcher has developed the following 
recommendations: 

• PMs should select the SDCE method when their only concern is to conduct a 
thorough evaluation of an offeror’s software development capability. This includes the 
evaluation of non-software issues that might affect the software development project. The 
overall goal is to reduce the software development risks to his program. 

• PMs should select the SDCE method when the SSA directs that the contract be 
awarded with "without discussions." The SDCE is the only method of the two that 
permits an evaluation of an offeror’s software development capability under "without 
discussion" conditions. 

• PMs should select the SCE method when there is concern with not only 
conducting an evaluation of an offeror’s software development capability, but also 
promoting process improvement. The overall goal would be to raise the maturity levels 
of the software industry in order to provide DOD with a mature supplier base capable of 
developing quality software within time and costs constraints. 


123 


• PMs should select the SCE method when their program resource constraints 
prohibit the funding of a SDCE, but can support the use of a SCE during source selection 
evaluation. 

• When faced with human resources constraints, PMs should select the method 
which best matches the availability of qualified/experienced personnel to the 
recommended evaluation team qualifications for the SDCE and SCE. 

D. AREAS FOR FURTHER RESEARCH 

The researcher encountered the following issues that are important but were outside 
the scope of this thesis: 

• What are the contractor’s costs to prepare and participate in a SCE or SDCE? 
How do these costs affect a contractor’s decision on whether or not to bid on a contract? 

• How may the SCE or SDCE be used for the purpose of monitoring a contractor’s 
performance? 

• How does DOD address the problem of a contractor being subjected to multiple 
software capability evaluations when bidding on several contracts, most of which 
incorporate the use of a software capability evaluation during source selection? 

• What are the advantages, disadvantages, and legal ramifications of providing SCE 
or SDCE teams with results of previous software capability evaluations to assist the 
evaluation team in preparing for a specific site visit? 

• If the IDA survey was repeated today, what would be the results? 
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• What is the correlation between a contractor’s maturity level and the results it 
achieves? 

• As a related issue, will an effective software developer have a high maturity level? 

• How does a contractor respond to the methods used to evaluate them? 

• Is it necessary that all KPAs be met before a given maturity level is realized? 

• Should criteria associated with high levels of maturity count in evaluating 
contractors at a lower target process capability? 

E. SUMMARY 

The differences in the SDCE and SCE evaluation methods are attributed to the goals 
of the organizations that created them. ASC/AFMC is committed to assisting PMs in the 
successful development of weapon systems. The SDCE is designed to identify and reduce 
development risks for embedded software with little regard for promoting process 
improvement within the software industry. The SEI's goal is to promote software process 
improvement. Use of the SCE during source selection evaluation, serves to motivate 
contractors to pursue CMM-based process improvements. The SCE has a price for 
promoting software process improvement, the scope of the evaluation is more narrow than 
that of the SDCE. As the SEI, and ASC/AFMC continuously work to improve their 
evaluation methods, the gap between identifying and reducing software development risks 
and promoting software process improvement may close. 
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